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1.0  Center  for  Advanced  Sensors 


1.1  Administrative 

This  report  covers  the  period  of  performance  from  31March2005  through  31August2006 
which  includes  the  no  cost  extension  of  FY2005  funded  work.  Year  two  (FY  2006  funds) 
were  in  place  by  3  lMay2006.  However,  there  is  no  overlap  of  reported  work. 

Researchers  at  the  University  of  Memphis  completed  year  one  effort  by  3  lMarch2006  as 
reported  in  prior  quarterly  reports.  Vanderbilt  University  is  just  now  starting  the  year  two 
funded  efforts  and  the  University  of  Alabama-  Huntsville  is  not  participating  in  year  two. 
Consequently,  all  of  the  work  reported  in  this  Final  Report  was  funded  solely  by  the 
FY2005  funds.  Therefore  this  report  covers  effort  toward  the  Year  One  Annual  Program 
Plan.  The  efforts  under  FY2006  funding  from  3  lMay2006  through  3 1  August2006  are 
reported  in  a  separately  submitted  first  quarter  report  for  FY2006  funds  and  are  referred 
to  as  Year  Two  activities.  All  of  this  was  discussed  with  Steven  Murrill  from  the  Army 
Research  Laboratory  (ARL),  Adelphi,  MD,  our  Cooperative  Agreement  Manager,  to  get 
his  agreement  to  this  reporting  process. 


1 . 2  Cen  ter  A  ctivities 

The  Director  met  frequently  with  Steven  Murrill,  to  discuss  year  1  activities  and  plans  for 
year  2.  Year  Two  plans  called  for  increasing  the  number  of  supported  researchers  at  The 
University  of  Memphis  to  five  and  eliminating  the  subcontract  with  the  University  of 
Alabama  Huntsville. 


1.2.1  Providing  Support  to  Research  Efforts 

Center  personnel  provided  continued  support  to  research  efforts  at  the  Army  Research 
Laboratory  in  the  areas  of  sensor  networks  and  Intelligence/Surveillance/Reconnaissance 
(ISR)  sensors.  This  activity  will  continue  as  part  of  Year  Two  efforts. 

The  Army’s  Night  Vision  and  Electronic  Sensors  Directorate  (NVESD)  was  supported  by 
frequent  contact  with  Dr.  Halford.  Support  was  primarily  for  the  Modeling  and 
Simulation  Division’s  research  activities. 

Dr.  Halford  continued  to  support  Dr.  Tim  Edwards  (under  separated  funding)  from  the 
Army’s  Redstone  Technical  Test  Center  (RTTC).  Human  perception  measurements  at 
the  Center  were  obtained  and  reported.  Interpretation  of  the  data  obtained  in  the 
perception  experiments  is  still  being  analyzed. 


1.2.2  Perception  Laboratory 

The  Center’s  Perception  Laboratory  continues  to  support  military  research.  The  NVESD 
perception  experiments  in  February,  May  and  August  were  conducted  at  The  University 
of  Memphis.  Additional  experiments  are  planned  for  November,  2006.  The  laboratory 
will  continually  evolve  but  a  complete  set  of  the  initially  planned  equipment  is  in  place. 
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The  sections  that  follow  present  a  summary  of  first  year  activity  on  each  of  the 
milestones  in  the  Annual  Program  Plan. 


1.3  Milestone  One:  Build  an  ontology-based,  knowledge 
repository  of  representative  sensors 

1.3.1  Purpose  of  Project  Milestone 

A  high  priority  of  the  U.  S.  Military,  and  in  particular  the  U.S.  Army,  is  the  requirement 
to  integrate  information  from  a  variety  of  heterogeneous  sensors  and  other  infonnation 
sources.  The  ability  to  discover  a  set  of  sensors  based  on  a  user’s  high-level  query 
specification  and  to  subsequently  configure  and  receive  data  from  these  sensors  for 
missions  that  may  not  have  been  anticipated  at  the  time  the  sensors  were  deployed  is 
critical.  Ideally,  such  information  integration  will  facilitate  the  construction  of  accurate 
situation  assessment  synthesized  from  shared  data.  Specifically,  according  to  [Scherl, 
2004]  the  Anny’s  Future  Force  will  carry  out  Network  Centric  Warfare,  which  is  defined 
by  [Alberts,  2002]  as  “an  information  superiority-enabled  concept  of  operations  that 
generates  increased  combat  power  by  networking  sensors,  decision  makers,  and  shooters 
to  achieve  shared  awareness,  increased  speed  of  command,  a  higher  tempo  of  operations, 
greater  lethality,  increased  survivability,  and  a  degree  of  self-synchronization.” 
Therefore,  the  motivation  of  this  project  milestone  was  to  investigate  techniques  by 
which  sensors  and  sensor  repositories  could  be  described  in  a  computer-readable  format, 
that  is,  the  development  of  an  ontology-based  knowledge  repository  of  sensors.  Such  a 
knowledge  base  could  be  used  to  enable  applications  to  locate,  access,  and  in  some  cases, 
facilitate  on-the-fly  tasking  of  sensors  in  an  automated  manner  via  computer  agents. 

1.3.2  Background:  Ontological  Engineering 

Since  the  1980s,  Ontological  Engineering  has  been  a  subject  of  computing  research, 
particularly  in  knowledge  representation  and  automated  reasoning.  The  work  in 
ontologies  centers  on  developing  shared,  machine  readable  conceptualizations  of 
knowledge  that  can  be  used  by  intelligent  computer  systems.  Representative  work 
concerning  ontologies  includes  [Lenat,  1989;  Skuce,  1990;  Uschold,  1996; 
Chandrasekaran,  1999]  and  more  recent  work  in  the  context  of  the  Semantic  Web 
includes  [Fensel,  2003;  McGuiness,  2003].  The  Semantic  Web  effort  [Berners-Lee, 
2001)  focuses  on  defining  and  moving  ontologies  and  knowledge  representation 
standards  from  traditional,  stand-alone  systems,  to  the  highly-distributed  World  Wide 
Web  (WWW).  Since  networked  sensors  were  our  focus,  the  evolving  infrastructure 
being  standardized  for  the  Semantic  Web  was  viewed  as  the  appropriate  means  for 
developing  our  sensor  ontology  and  laboratory  prototype  environment.  Much  work  in 
classical  Artificial  Intelligence  research  is  seeing  renewed  application  in  the  context  of 
the  Semantic  Web  [Russomanno,  2003]. 

As  stated  by  [Llinas,  2004]:  “One  main  benefit  typically  cited  for  basing  the  development 
of  an  information  process  on  an  ontological  footing  is:  interoperability,  with  other  local 
and  also  external  processes,  which  leads  to  a  shared  understanding.”  Therefore,  our  year 
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one  investigation  has  focused  in  part  on  the  following:  i)  how  an  ontology  of  sensors  can 
be  defined  and  used  to  enable  processes  that  require  sensor  interoperability  and  fusion; 
and  ii)  the  development  of  a  prototype  environment  for  proof-of-concept  use  of  the 
ontology. 

1.3.3  Accomplishments 

As  part  of  CAS  knowledge  engineering  year  1  activity,  we  have  conducted  a  thorough 
literature  survey  related  to  sensor  ontologies  and  approaches  to  sensor  service 
specifications.  We  have  implemented  a  computer-readable  ontology  and  accompanying 
prototype  environment.  We  have  outlined  an  approach  for  locating  sensors  by  semantic 
processing  of  an  agent’s  query.  Details  of  these  accomplishments  are  described  in  the 
following  paragraphs. 

Computer-Readable  Ontology 

We  developed  a  computer-readable  sensor  ontology  and  have  coined  it  as  OntoSensor 
[Russomanno,  2005],  This  work  deviated  somewhat  from  the  plans  stated  in  the  initial 
project  proposal  to  focus  primarily  on  defining  imaging  sensors  in  the  ontology.  This 
change  in  direction  was,  in  part,  because  of  the  relative  simplicity  of  the  sensor  nodes  that 
were  readily  available  from  commercial  vendors  such  as  Crossbow,  Inc.  [Crossbow, 
2006]  as  compared  to  the  limited  availability  of  complex  imaging  sensors.  As  such,  the 
knowledge  engineering  activity  focused  on  creating  ontological  infonnation  for  low-cost 
wireless  sensor  network  configurations  to  gain  familiarity  with  sensor  knowledge 
modeling  techniques  and  available  tools,  before  moving  on  to  ontological  modeling  of 
complex  imaging  sensors.  These  simpler  wireless  sensors  typically  provide  scalar 
samples  rather  than  high-bitrate  data  sources  such  as  video  sensors  or  infrared  (IR) 
cameras.  In  addition,  capturing  ontological  information  for  these  simpler  sensors  and 
motes  enabled  the  rapid  development  of  a  prototype  environment  and  facilitated  graduate 
students  to  experiment  with  novel  approaches  for  the  design  of  software  agents,  sensor 
service  brokers,  and  the  utilization  of  Semantic  Web  infrastructure. 

Graduate  students  K.  Kothari,  O.  Thomas,  and  C.  Goodwin  (Kothari  and  Goodwin 
received  partial  support  from  CAS  funds)  made  contributions  to  OntoSensor,  including 
the  instantiation  of  a  variety  of  wireless  sensor  instances  into  the  knowledge  base.  Our 
approach  was  that  the  specific  sensor  instances  were  to  be  specified  in  distributed 
repositories  located  at  base  stations  throughout  a  network,  while  OntoSensor  resided  on  a 
central  server  and  contained  categorical  descriptions  of  the  various  sensors  including 
their  attributes  and  associations.  A  screen  shot  of  OntoSensor  is  provided  in  Figure  1.1- 
1. 

All  sensors  used  in  our  prototype  environment  in  laboratory  Engineering  Science  (ES) 
223  have  associated  computer-readable  OntoSensor  definitions.  OntoSensor  provides  a 
basic  knowledge  base  of  various  sensor  types.  The  contents  of  OntoSensor  can  be 
exported  to  OWL  [Smith,  2003]  format  and  processed  by  any  software  with  OWL 
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parsing  capability.  The  export  of  OntoSensor  is  accessible  via  the  following  URL: 
http://www.eece.memphis.edu/cas/OntoSensor/OntoSensor.owl. 

OntoSensor  also  includes  knowledge  models  for  a  variety  of  data  acquisition  boards, 
sensing  elements,  and  processor/radio  units,  as  well  as  preliminary  definitions  for  a  few 
imaging  sensors.  OntoSensor  concepts  and  associations  are  instantiated  in  distributed 
repositories  updated  by  the  base  stations  of  the  network.  The  data  model  for  a  given 
sensor  provides  containers  for  meta  data  such  as  sensitivity,  performance  range,  and 
accuracy  for  the  sensing  elements,  as  well  as  physical  characteristics  such  as  mass,  radio 
frequencies,  dimensions,  and  power  supply  information  for  wireless  motes. 
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Figure  1.1-1:  Screen  shot  of  the  OntoSensor  interface  using  Protege  2000 


We  also  developed  code  for  some  basic  queries  of  OntoSenor  using  SWI-Prolog  libraries 
[Wielemaker,  2003]  to  illustrate  a  concrete  approach  of  how  the  ontology  could  be 
accessed.  The  code  is  independent  of  the  ontology  but  can  read,  parse,  and  perfonn 
inference  on  OntoSensor,  which  has  been  exported  to  OWL  and  posted  to  the  Web. 


Laboratory  Prototype 


The  laboratory  prototype  implementation  focused  primarily  on  using  ontological  models 
of  representative  sensors  commonly  found  in  wireless  network  applications  and  the 
definition  of  an  architecture  for  locating  sensor  services  with  capabilities  and  other 
properties  that  satisfy  a  user’s  query.  This  laboratory  prototype  effort  included  the 
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preparation/submission  of  a  paper  and  poster  to  the  IEEE  Fifth  International  Conference 
on  Information  Processing  in  Sensor  Networks  [Goodwin,  2006].  This  paper  describes 
work-in-progress  development  of  our  ontology-based,  sensor  network  prototype 
environment.  The  majority  of  the  sensors,  motes,  and  base  stations  used  in  the  prototype 
were  purchased  from  CAS  funds. 

The  wireless  environment  consists  of  two  networks,  each  having  multiple  sensor  nodes 
built  from  parts  available  from  commercial  vendors  as  shown  in  Figure  1.1-2.  Sensor 
types  include  models  MEP410,  MEP510,  MTS420,  and  MTS310  available  from 
Crossbow  Inc.  (examples  are  shown  in  Figure  1.1-3).  The  MEP410  and  MEP510  are 
environmentally  hardened  mote  platforms.  The  MEP410  platform  sensing  capabilities 
include  ambient  light,  barometric  pressure,  photo-sensitive  light,  relative  humidity,  and 
temperature.  The  MEP5 10  platfonn  is  capable  of  relative  humidity  and  temperature 
measurements.  A  MIB5 10  gateway  is  used  for  the  aggregation  of  data  obtained  from  the 
MEP410  and  MEP5 10  platforms.  Also,  MTS420  and  MTS3 10  data  acquisition  sensing 
elements  are  coupled  with  MICAz  processor/radio  boards.  The  MTS420  sensor  boards 
have  capabilities  that  include  bi-axial  accelerometer,  ambient  light,  barometric  pressure, 
temperature,  GPS,  photo-sensitive  light,  and  relative  humidity  measurements.  The 
MTS310  sensor  boards  have  bi-axial  accelerometer,  bi-axial  magnetometer,  acoustic, 
temperature,  and  photo-sensitive  light  sensing  elements.  The  MTS420  and  MTS310 
deployment  uses  a  MIB520  network  gateway  which  collects  data  from  remote 
deployments.  The  MICAz  boards  operate  in  the  916  MHz  or  2.4  GHz  frequency  range 
and  handle  the  data  processing  and  communication  needs  for  the  platform.  The 
processor/radio  boards  run  Crossbow’s  XMesh  software  that  allows  dynamic  formation 
of  communication  links  between  the  nodes. 


Sensor 

Instances  and/or 


Figure  1.1-2:  Prototype  Sensor  Network  Constructed  in  Engineering  Science  (ES)  223 
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Figure  1.1-3:  Example  wireless  sensor  nodes  and  motes  used  in  the  development  of  the 
prototype  environment. 


Currently,  two  computers  interface  with  the  associated  base  stations  using  Crossbow’s 
MoteView  application  to  retrieve  data  which  is  stored  in  a  PostGRE  database.  Each  base 
station  executes  custom  software  developed  by  C.  Goodwin  that  generates  OntoSensor 
referenced  OWL  repositories  from  the  collected  sensor  data.  The  software  is  a 
preliminary  implementation  of  a  web  service  that  will  be  evoked  at  the  base  stations  for 
exchanging  OWL  data  within  the  network  environment. 

The  software  executing  on  each  base  station  periodically  checks  for  a  database  update 
using  PostGRE  ODBC  drivers  and  retrieves  updated  tuples  corresponding  to  raw  sensor 
percepts.  The  retrieved  tuples  are  then  post-processed  and  marked  up  using  OWL  and 
made  available  for  potential  data  sharing  with  other  processes. 

Additional  laboratory  activity  included  experimenting  with  the  various  Crossbow  Inc. 
sensors  and  motes  to  gain  an  understanding  of  their  sensing  capabilities,  programmable 
features  and  suitability  for  potential  ontology-based  proof-of-concept  applications  within 
the  context  of  our  prototype  environment. 

Literature  Survey 

We  performed  an  extensive  literature  survey  for  related  efforts.  Our  survey  indicates  that 
although  OntoSensor  is  somewhat  superficial  in  content,  it  is  considerably  more  detailed, 
for  a  given  sensor  type,  than  other  sensor  ontologies  that  are  freely  available.  For 
example,  OntoSensor  provides  more  details  about  the  sensor  types  and  instances  defined 
in  its  knowledge  base  than  the  SWEET  sensor  ontology  [Raskin,  2006]  provided  by  the 
Jet  Propulsion  Lab  accessible  via  the  following  URL:  http://sweet.jpl.nasa.gov/sweet. 
SWEET  may  provide  more  sensor  types  than  OntoSensor,  but  provides  no  computer- 
readable  details  about  the  various  sensor  types. 

We  also  monitored  and  analyzed  the  Open  Geospatial  Consortium’s  evolving 
specification  of  SensorML.  SensorML  is  part  of  the  SensorWeb  enablement  effort  which 
shares  many  of  the  same  goals  of  our  work  in  sensor  interoperability,  discovery,  on-the- 
fly  tasking,  etc.,  but  this  effort  does  not  currently  employ  Semantic  Web  infrastructure. 
Moreover,  many  of  the  class  diagrams  provided  in  the  SensorML  specification  are 
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generic  and,  when  used  alone,  lack  the  detail  and  semantics  for  creating  a  sensor 
ontology.  However,  the  possibility  of  augmenting  SensorML  specifications  with 
concepts  from  one  or  more  fonnal  ontologies  may  be  a  promising  approach. 

Our  design  of  OntoSensor  was  based  (in-part)  upon  a  draft  SensorML  specification  [OGC 
04-019,  2004].  The  2004  SensorML  specification  has  subsequently  undergone 
substantial  revisions  [OGC  05-086r2,  2006].  Nevertheless,  our  initial  approach  to  the 
implementation  of  OntoSensor  is  continuing  to  be  used  in  our  prototype  environment. 
However,  there  is  an  opportunity  and  need  to  re-organize  OntoSensor  to  better 
complement  the  revised  SensorML  specification,  which  may  provide  wider  visibility, 
utility,  and  adaptation  of  our  approach  to  sensor  ontology  realization. 

Sensor  Services 

We  investigated  various  approaches  for  defining  and  implementing  sensor  services  for 
the  prototype  environment  such  that  an  agent  can  locate  services  based  on  a  high-level 
query  and  subsequently  receive  data  from  and/or  task  the  sensors.  A  service-oriented 
architecture  [Singh,  2005],  in  which  sensors  publish  their  availability  to  a  sensor  service 
broker  that  provides  a  type  of  yellow-pages  description  of  sensor  services,  can  be  queried 
by  a  sensor  service  requestor  to  locate  sensors  having  certain  properties  that  might  satisfy 
both  a  geographic  as  well  as  capability  constraints.  A  generic  model  for  sensor  services 
is  shown  in  Figure  1.1-4.  Sensors  have  corresponding  service  definitions  (function 
declarations  that  are  accessible  over  the  network)  that  include  binding  information  such 
that  a  requestor  can  query  and/or  task  the  sensor,  sensor  network,  or  base  station.  In  our 
approach,  the  sensor  service  providers  are  the  actual  individual  sensors,  sensor  networks 
or  base  stations  that  register  their  respective  services  with  the  broker  and  publish  their 
associated  OWL  repositories  on  the  network. 

Both  Sun  Microsystems  [Horan,  2005]  and  Microsoft  [Liu,  2005a;  Woo,  2005;  Liu, 
2005b]  have  basic  research  efforts  related  to  this  area.  In  addition,  the  Naval  Research 
Laboratory  (NRL)  [Luo,  2005]  defines  an  approach  to  support  semantic  service 
description  and  match  making  with  registries  that  use  the  existing  Universal  Description, 
Discovery  and  Integration  (UDDI)  V3  specification. 


{sensor  network(s)  {software  agent(s)} 

&  base  station(s)} 

Figure  1.1-4:  Generic  Model  for  Sensor  Services 


Our  activity  focused  on  the  matching  process  of  finding  the  appropriate  sensors  for  a 
given  high-level  query  using  knowledge-based  support  that  is  not  part  of  the  sensor 
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service  broker’s  registry.  Unlike  the  NRL  approach  taken  by  [Luo,  2005],  which 
provides  a  method  to  bulk-load  semantic  data  into  UDDI  TModels,  our  activity  focused 
on  developing  an  approach  to  associate  OntoSensor  with  a  UDDI  registry  of  sensor 
services  to  support  semantic  searching.  A  simplification  of  our  approach  is  shown  in 
Figure  1.1-5.  In  this  scheme,  the  UDDI  database  can  be  viewed  as  a  sensor  services 
cache.  If  the  agent’s  query  results  in  hits,  that  is,  sensor  services  that  match  the  user’s 
request  are  directly  specified  in  the  UDDI  registry,  then,  these  sensors’  bindings 
(specifications  to  query  and/or  task  the  sensor)  are  returned  to  the  agent;  otherwise,  the 
UDDI  sensor  service  interface  will  attempt  to  find  generalizations  or  specializations  via 
OntoSensor  that  may  semantically  match  (to  some  degree)  the  agent’s  query.  If  such 
matches  are  found,  then,  they  will  be  directly  updated  in  the  UDDI  registry,  much  like  a 
conventional  cache,  for  rapid  retrieval  of  subsequent  solutions  to  agent’s  queries. 


Sensors  ■ 


Sensor  Service  Broker 


Generalization  of  Sensor? 
Specialization  of  Sensor? 
Instance  of  a  Sensor  Class?A 


Semantic 

Matcher 


2.  Semantic  search  for 
sensors  that  may  satisfy 
agent's  request 


UDDI  Sensor 
Service  Interface 


Query  Processor 
(Prolog) 


locate 


publish 


1 .  Exact  Match? 

-  Yes  (hit)  return  sensor(s) 

-  No  (miss)  use  OntoSensor 
to  determine  if  sensor(s)  with 
similar  capabilities  are  available 
within  the  agent's  constaint 
(step  2) 

3.  Insert/Update/Delete 
Semantic  Information 


Agent - 

(SWEXSYS/Prolog) 


task  & 
query 


Figure  1.1-5:  Sensor  semantic  service  description  and  match  approach  for  prototype 


1.3.4  Future  Work 

OntoSensor  and  Prototype  Enhancements 

As  previously  reported,  it  is  possible  that  OntoSensor  may  have  the  potential  for  wider 
adoptability,  contributions,  and  assessment  from  a  variety  of  user  communities  if  it  is 
reorganized  to  better  complement  the  revised  OGC  SensorML  specification.  Knowledge 
reorganization  of  OntoSensor  is  needed  in  year  2,  while  continuing  to  use  the  current 
model  in  the  prototype  environment  to  support  development  of  the  sensor  service  and 
agent  functionality. 
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To  support  a  variety  of  possible  tests  and  proof-of-concept  applications,  future  prototype 
enhancements  require  the  inclusion  of  more  complex  sensors  into  OntoSensor  and  the 
network  including  imaging  sensors  such  as  video,  visible,  infrared,  terahertz  and  other 
sensors  that  provide  rich,  high-bitrate  data.  For  example,  a  ThermaCam  Merlin  mid¬ 
wave  infrared  camera  (or  similar  camera)  and  other  CCD  cameras  are  currently  pending 
integration  into  the  prototype  environment.  Cameras  such  as  Merlin  connect  directly  to 
the  LAN  and  include  a  software  development  toolkit  (SDK).  Ideally,  we  would  use  the 
SDK  to  develop  service  descriptions  such  that  the  cameras  could  be  queried  and  tasked 
by  agents  that  locate  the  cameras’  services  via  the  broker  without  requiring  detailed 
knowledge  of  the  sensors’  device  drivers  or  API. 

The  SDK  will  also  be  used  for  obtaining  periodic  frames  and  marking  up  knowledge 
about  captured  images  in  OWL.  These  images  and  their  associated  OWL  repositories 
will  be  stored  at  the  associated  base  stations  that  can  be  subsequently  discovered  and 
queried  by  agents  and  end  users.  The  OWL  repositories  will  contain  meta  data  that 
includes  image  fonnat,  date,  time,  resolution,  location  and  other  properties  about  the 
sensor  that  captured  the  images  and  the  surrounding  environment.  The  meta  data 
contained  in  OntoSensor  describing  cameras  may  be  used  to  deduce  further  information 
about  their  acquired  image.  Furthermore,  a  time  series  of  images  can  be  retrieved  from  a 
distributed  repository  and  post-processed  to  extract  intrinsic  knowledge  that  can  be 
combined  with  scalar-level  readings  such  as  temperature,  vibration,  acoustic,  magnetic 
field  measurements,  or  other  information  sources  to  potentially  increase  the  confidence  in 
a  target  or  activity  of  interest. 

Additional  work  is  needed  to  further  implement  the  sensor  service  architecture  outlined  in 
Figure  1.1-5.  Currently,  the  agent  is  only  capable  of  examining  a  simple  database  of 
URLs  to  locate  and  load  sensor  repositories  from  the  various  base  stations  into  its 
knowledge  base  to  perform  simple  queries  and  reasoning  processes.  The  ability  for  the 
agent  to  perfonn  semantic  searching  and  selection  and  the  capability  to  combine  web 
services  using  the  scheme  outlined  in  Figure  1.1-5  is  desired. 

OntoSensor  may  be  further  extended  using  open  standards  provided  by  OWL-S  [Martin, 
2004]  to  capture  higher-level  knowledge  about  sensor  services.  From  the  view  of  the 
sensor  service  requestor  (agent),  the  deployed  networks  will  appear  as  a  seamless 
environment  that  provides  a  set  of  services.  Realization  of  this  enviromnent  will  require 
implementation  of  a  comprehensive  sensor  service  broker. 
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1.4  Performance  modeling  of  sensors  with  image  processing 
enhancements 

The  use  of  image  processing  enhancements  in  sensors  is  now  widely  accepted.  The 
techniques  include  ATRs,  image  fusion,  local  area  processing  (such  as  local  contrast 
enhancement),  image  stabilization,  super  resolution  (primarily  reconstruction  with 
undersampled  imagers)  and  turbulence  mitigation.  Other  techniques  such  as  boost  and 
logarithmic  amplification  can  also  be  considered  although  they  are  not  necessarily  done 
by  processing  the  imagery.  Color  presents  another  opportunity  for  image  processing 
enhancement.  Color  can  be  coupled  with  image  fusion  to  present  the  fused  image  in  an 
optimum  manner. 


1.4.1  Milestone:  Establish  methodology  for  incorporating  image 
processing  techniques  into  existing  performance  models. 

Each  of  the  techniques  has  been  studied  to  determine  an  effective  means  of  incorporating 
their  effects  into  a  sensor  model.  The  status  of  this  milestone  is  summarized  by 
considering  each  technique. 

Image  stabilization  in  effect  reduces  the  blur.  Its  incorporation  into  a  sensor  model  is 
therefore  accomplished  by  determining  the  equivalent  reduction  in  motion  of  the  imager. 
Blur  due  to  motion  is  well  understood  so  the  effect  of  its  reduction  is  readily  incorporated 
into  a  sensor  model. 

Local  contrast  enhancement  can  also  be  readily  incorporated  into  sensor  models.  The 
local  area  contrast  rather  than  scene  contrast  is  used  to  detennine  human  performance. 

Turbulence  mitigation  can  similarly  be  incorporated  by  determining  an  effective  value  of 
Cn2,  the  structure  constant  for  the  index  of  refraction.  Much  work  remains  for  this 
technique,  but  its  incorporation  into  the  sensor  model  is  now  clear. 

Super  resolution  is  treated  in  a  similar  manner,  except  that  the  technique  affects  two 
sensor  parameters.  The  performance  improvement  is  mapped  into  a  resolution/noise 
space.  For  example,  the  process  is  known  to  decrease  noise  and  increase  resolution.  This 
provides  a  means  of  quantitatively  expressing  the  improvement  afforded  by  the 
processing.  Both  of  these  parameters  impact  human  discrimination  in  well  studied 
manners.  Consequently,  super  resolution  algorithms  are  readily  modeled  as  changes  in 
resolution  and  noise.  While  this  was  reported  in  the  first  quarter  report  it  is  repeated  here 
because  it  has  been  used  and  seems  to  provide  satisfactory  results. 

Modeling  ATRs  remains  an  active  area  of  work  within  the  Center.  Specifically,  the 
anomaly  detection  algorithm  reported  by  Singer  is  the  ATR  being  modeled  [Singer, 

1999].  However,  our  work  has  identified  two  means  of  incorporating  the  ATR 
performance  into  the  sensor  model.  They  differ  in  their  methods  of  obtaining  the  signal- 
to-clutter  (SCR)  ratio.  One  technique  requires  an  image  whereas  the  second  uses 
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previously  determined  parameters  from  typical  imagery  to  characterize  the  power  spectral 
density  of  the  background.  It  also  requires  spectral  properties  of  representative  materials 
such  as  paints,  grass,  asphalt,  bricks  and  cinder  block.  This  model  represents  an  original 
contribution  that  will  form  the  basis  of  a  journal  submission.  It  is  considered  the  best 
way  to  incorporate  ATR  models  into  a  sensor  mode.  This  is  because  most  sensor  models 
are  physics-based  models  and  do  not  utilize  imagery  as  an  input.  More  detail  is  provided 
in  Sections  1.4.2.  The  emphasis  of  the  model  is  for  infrared  imagers,  primarily  those 
operating  in  the  mid  wave  and  long  wave  IR. 

The  SCR  can  be  considered  a  generalization  of  the  signal  to  noise  ratio  (SNR),  often  used 
to  quantify  the  performance  of  a  single  band  imaging  system.  The  SNR  is  simply  the 
magnitude  of  the  target  divided  by  the  standard  deviation  of  the  background.  SCR 
generalizes  SNR  to  handle  multiple  spectral  bands  and  spatial  processing. 

Once  the  SCR  is  estimated,  receiver  operating  characteristic  (ROC)  curves  can  be 
obtained  for  some  algorithms.  For  the  Singer  algorithm  [Singer,  1999],  the  SCR 
determines  the  non-centrality  parameter  for  the  non-central  F  distribution.  Basically,  the 
cumulative  distribution  function  of  the  non-central  F  distribution  is  the  ROC  curve  for  the 
ATR.  As  always,  such  a  result  depends  on  modeling  assumptions.  For  example,  an 
assumption  of  Gaussian  statistics  for  pixel  distribution  leads  to  the  non-central  F 
distribution.  It  is  apparent  that  Gaussian  statistics  may  not  represent  the  tails  of  the 
distribution  well.  And,  the  tails  of  the  distribution  are  the  critical  pixels  for  a  detection 
process  where  atypical  pixels  are  being  sought.  However,  such  assumptions  are  likely  to 
have  little  impact  on  comparative  uses  of  the  model,  one  sensor  vs.  another,  one  band  vs. 
two,  different  spectral  bands  (band  selections).  Furthennore,  the  SCR  is  likely  the  critical 
parameter  for  almost  any  ATR  algorithm,  even  if  the  ATR  algorithm  does  not  lend  itself 
to  analytical  modeling. 


1.4.2  Milestone:  Determine  sensor  performance  enhancement 
offered  by  anomaly  detection  ATRs. 


1.4.2. 1  Algorithm  Description  for  SCR  Calculation 

The  steps  of  the  algorithm  for  estimating  a  SCR  for  a  sensor  are  enumerated  below.  They 
indicate  some  of  the  infonnation  required  as  well  as  what  sensor  aspects  are  treated. 

Most  of  the  input  parameters  come  from  models  obtained  from  measured  data.  The 
actual  ATR  algorithm  is  run  to  calculate  the  final  SCR.  This  provides  a  means  to  perhaps 
generalize  the  model  to  many  different  algorithms.  That  generalization  was  not 
developed.  The  Singer  algorithm  [Singer,  1999]  was  chosen  as  representative  of  the 
performance  that  could  be  obtained  with  an  anomaly  detection  algorithm. 

The  model  calculations  are  performed  in  several  separate  steps  to  generate  the  required 
input  data  and  calculate  intennediate  results.  These  divisions  are: 

1 .  Optics  calculations 

2.  Background  spatial  shape 
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3.  Background  spatial  autocorrelation 

4.  Target  spatial  shape 

5.  Sensor  spectral  responses  to  target  and  background 

6.  Spectral  covariance 

7.  Noise 

8.  Target  vector 

9.  Background  vectors  (forms  a  matrix) 

10.  Clutter  covariance  matrix 

11.  Target  and  background  response  vectors  and  noise  response 

12.  Spectral  anomaly  detector  matrix 

13.  Output  SCR 

Each  of  these  steps  forms  a  block  of  the  overall  ATR  (SCR)  model. 

The  SCR  is  a  measurement  of  the  amount  of  independent  information  about  a  target 
obtained  from  a  sensor  in  both  the  spectral  and  spatial  domains.  Adding  additional 
spectral  bands  increases  the  amount  of  raw  data  collected,  but  the  amount  of  useful 
information  only  increases  if  the  new  bands  return  some  uncorrelated  data.  Similarly, 
spatial  filtering  will  give  more  information  about  the  locations  of  possible  targets  only 
when  the  target  shape  and  the  clutter  shapes  are  different. 

The  SCR  measures  the  relative  amount  of  information  available  upon  which  to  perfonn 
target  detection  with  a  space-spectral  anomaly  detector  style  ATR.  Increasing  the 
amount  of  input  information  is  assumed  to  increase  the  final  performance  of  a  general 
ATR  algorithm.  This  model  does  not  model  any  specific  ATR  implementation  or  any 
additional  processing  such  as  target  trackers. 


1.4.2.2  Spectral  Radiance  Model 

The  model  requires  a  spectral  model  of  the  target  and  background.  One  such  model  is 
presented.  It  uses  planar  surfaces  in  the  form  of  a  box  sitting  on  a  plane  and  located  near 
a  planar  wall.  The  box  is  a  tank-sized  box  and  the  wall  can  represent  a  building  or  a  tree 
line,  depending  on  what  materials  are  specified.  This  model  is  intended  to  give  quick  and 
useful  spectral  target  radiance  predictions  without  a  high  level  of  expertise  in  thennal 
prediction  being  required  by  the  user.  This  model  is  an  effective  tool  for  comparisons  of 
different  sensors  or  for  band  selection. 

To  simplify  the  calculation  of  spectral  radiance,  some  assumptions  have  been  made  about 
the  materials  and  surfaces.  All  materials  and  surfaces  are  assumed  to  be  opaque  and 
Lambertian.  The  opaque  assumption  avoids  the  need  to  handle  translucent  materials. 
Assuming  opaque  materials  allows  the  material  properties  to  be  fully  specified  by  either 
reflectance  or  emittance.  Very  few  materials  are  translucent  in  the  infrared  bands.  The 
Lambertian  assumption  allows  geometric  calculations  to  be  separated  from  spectral 
calculations.  This  greatly  reduces  the  amount  of  calculation  necessary.  Most  materials 
are  close  to  Lambertian  in  the  infrared.  Handling  non-Lambertian  surfaces  would  greatly 
complicate  the  model. 
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A  single  target  and  background  geometry  with  4  look  angles  has  been  chosen  to  simplify 
the  spectral  radiance  model.  The  target,  wall,  and  ground  and  the  4  view  angles  are 
shown  in  Figure  1 .4.2-1.  The  assumed  target  consists  of  a  roughly  tank-size  box  6m 
long,  3m  wide,  and  2m  high.  It  is  located  10m  away  from  a  12m  high  wall  of  essentially 
infinite  length.  The  4  views  are  0°,  0.1  radian  declination,  45°  declination,  and  90° 
declination.  The  0°  and  0.1  radian  views  are  typical  of  ground-to-ground  engagements. 
The  45°  view  is  intended  to  model  an  Unmanned  Aerial  Vehicle  (UAV)  view.  The  90° 
view  models  UAV,  aircraft,  or  satellite  views. 

M  I  /  \  X  \  \A  \  / 


Figure  1.4.2-1:  box  and  wall  geometry 

The  radiometric  parameters  for  this  modeled  have  been  worked  out  through  numerical 
integrations,  validated  by  limiting  cases  for  which  analytical  solutions  exist.  MODTRAN 
can  be  used  in  conjunction  with  this  model  to  obtain  the  effects  of  sky  radiance  and  solar 
illumination.  With  the  spectral  characteristics  of  the  materials,  one  can  obtain  the 
radiance  for  the  sensor  with  any  of  the  chosen  orientations. 


1.4.2.3  Model  Validation 

Numerical  validation  is  difficult  to  obtain.  Instead,  the  model  is  validated  for  trends  and 
tendencies.  The  model  predicts  that  for  certain  choices  of  materials  and  bands,  two  bands 
provide  a  higher  SCR  than  a  single  band.  Similarly,  three  bands  (properly  chosen  for 
materials  assumed)  provides  a  higher  SCR  than  two  bands.  The  same  prediction  is 
available  from  analysis  of  actual  imagery  and  from  actual  ATR  algorithms. 

It  is  not  a  general  truth  that  two  bands  are  always  better  than  one  or  that  three  bands  are 
always  better  than  two.  This  two  is  borne  out  in  the  model  predictions. 
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The  details  of  the  model  are  being  prepared  for  submission  to  a  suitable  journal  for 
publication. 


1.4.3  Milestone:  Modify/adapt  existing  sensor  models  to  account  for 
the  improvements  offered  by  image  fusion. 

This  activity  is  still  in  process  and  expected  to  remain  an  active  area  into  year  two  of  this 
grant.  Progress  has  been  made  in  establishing  a  process.  However,  the  process  involves 
extensive  perception  laboratory  testing  to  quantify  the  improvement.  The  use  of  color 
has  generated  considerable  interest.  Consequently,  color  models  were  studied  and 
significant  results  are  the  subject  of  a  paper  being  submitted  for  publication  [Krapels, 
2007]. 

The  details  of  the  fusion  work  accomplished  to  date  are  provided.  Perception 
experiments  are  scheduled  for  August,  2006  to  provide  some  validation/calibration  of  the 
metrics.  More  imager  may  become  available  leading  to  additional  perception 
experiments  if  warranted.  Otherwise,  this  effort  may  be  concluded  by  the  mid  point  of 
the  second  year  of  Center  activities. 


1.4.3. 1  Fusion  Introduction 

Several  fusion  algorithms  are  discussed.  The  described  fusion  methods  are  applied  to 
LWIR  and  MWIR  images  of  urban  scenes.  Several  image  fusion  metrics  are  described  as 
well  and  applied  to  analyze  images  fused  using  the  reviewed  techniques.  The  results  of  a 
paired  comparison  perception  test  are  then  analyzed  to  see  in  what  order  the  images  are 
preferred  by  observers.  The  observer  preference  is  then  compared  with  the  fusion  metrics 
to  see  which  metric  best  correlates  to  the  observer  preference. 

Despite  the  very  active  research  in  computer  vision,  many  image  related  tasks  are  best 
performed  by  human  observers.  It  is  also  noted  that  images  from  different  spectral  bands 
can  often  provide  complimentary  infonnation.  In  order  to  fill  the  human  visual 
information  channel,  there  is  a  need  to  fuse  images  from  different  imagers.  The  hope  is 
that  successfully  fused  imagery  will  provide  to  the  observer  with  the  important 
information  from  all  of  the  source  images  while  causing  the  least  possible  amount  of 
distortion,  noise,  or  image  quality  degradation.  For  fusion  to  be  worth  while,  fused 
images  must  allow  for  better  observer  performance  than  either  of  the  source  images. 

There  are  many  issues  that  must  be  addressed  for  successful  image  fusion  to  take  place. 
First,  the  images  to  be  fused  must  be  registered.  The  imagers  should  have  similar  fields 
of  view  and  detector  resolution.  The  images  also  should  be  obtained  sufficiently  close  to 
one  another  both  spatially  and  temporally.  If  these  conditions  are  not  met,  fusion  may 
not  be  possible.  Different  tasks  may  also  dictate  how  much  time  and  computing  power  is 
available  for  fusion  to  take  place.  Another  issue  is  whether  or  not  fused  issues  should  be 
displayed  as  monochromatic  or  color  images.  Toet  [Toet,  2003]  has  shown  that 
appropriate  color  maps  can  be  beneficial  for  task  perfonnance  using  fused  imagery  and 
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better  fill  the  visual  channel.  He  also  observed  that  inappropriate  color  maps  actually 
cause  observer  performance  to  drop  below  that  possible  with  monochromatic  images.  In 
this  report  we  will  report  results  from  a  perception  experiment  using  grayscale  fused 
imagery.  Another  issue  is  that  noise  can  be  introduced  in  the  fused  image  through  either 
of  the  source  images  or  by  the  fusion  technique  itself.  Since  the  visual  channel  is 
bandlimited,  image  fusion  often  requires  that  some  information  in  the  source  images  be 
attenuated  or  eliminated.  The  different  fusion  algorithms  reviewed  here  show  different 
attempts  to  select  and  combine  the  relevant  information  from  source  images  into  a  single 
fused  image. 


1.4.3.2  Fusion  Techniques 

Pixel  by  pixel  averaging  is  the  most  common  of  the  image  fusion  techniques.  It  is  simple 
and  can  be  implemented  in  real-time  with  both  analog  and  digital  systems.  Averaging 
does  not  perform  well  when  images  to  be  fused  have  areas  of  opposing  contrast.  This 
destructive  interference  can  result  in  a  fused  image  with  decreased  contrast  for  important 
features  visible  in  both  of  the  image  bands. 

Gabor  image  fusion  uses  a  concept  from  psychophysical  data  which  suggests  that  human 
observers  weight  the  visual  infonnation  they  receive  based  on  spatial  frequency  and 
orientation.  In  the  spatial  domain  the  Gabor  functions  can  be  represented  in  this  form: 

g(x,y )  =  K  exp(-  7i{a(x  -  x0f  +  b(y  -  y0)2))exp(y(2^(w0x  +  v0y)  +  P))(1.4.3.1) 

In  the  equation  above,  K  detennines  the  magnitude  of  the  envelope.  The  terms  a  and  b 
scale  the  axes  of  the  Gaussian  envelope.  X0  and  yo  give  the  location  of  greatest  intensity 
in  the  Gaussian  envelope.  P  represents  the  phase  of  the  sinusoid.  Together  uo  and  \’o 
determine  the  frequency  of  the  sinusoid.  In  the  frequency  domain  the  Gabor  functions 
can  be  represented  as  filters  that  are  the  products  of  wedges  and  annuli. 

In  Gabor  fusion,  each  of  the  source  images  is  passed  through  a  bank  of  Gabor  filters  that 
divide  the  image  into  bands  depending  on  spatial  frequency  and  orientation.  For  a  given 
Gabor  filter  band,  the  magnitude  of  the  transfonn  of  each  image  is  compared.  The  image 
with  the  greatest  magnitude  in  the  band  is  then  used  in  the  construction  of  the  fused 
image.  In  this  way  the  most  important  parts  of  each  source  image  are  combined  to 
produce  the  fused  image. 

The  Discrete  Wavelet  Transfonn  image  fusion  method  is  another  technique  that  involves 
dividing  the  source  images  into  frequency  bands  and  then  constructing  a  fused  image 
using  source  image  bands  with  the  highest  magnitudes.  The  difference  between  the  DWT 
and  Gabor  techniques  lies  in  the  way  that  the  images  are  divided  into  bands.  The  DWT  is 
given  below: 
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i  M-lN-l 

w,‘  ( j ,  m,  n)  =  -J=  X  S/(x’  y^'j.m.n  (*,  y) 

i  M-lN-l 

K  U0,>n,n)  =  -^Y^f(x,y)<Pj^(x,y) 


(1.4. 3. 2) 


In  the  above  equations,  vj/  is  the  wavelet  basis  function,  and  cp  is  the  scaling  basis 
function.  W,,(  and  W<p  are  the  wavelet  and  scaling  approximations  of  the  image. 

The  Shift-Invariant  Discrete  Wavelet  Transform  fusion  technique  addresses  the  fact  that 
the  DWT  is  not  shift  invariant  due  to  the  downsampling  that  it  uses  in  its  filtering 
process.  This  can  cause  visible  artifacts  in  DWT  fused  images  that  are  not  perfectly 
registered.  To  overcome  this  problem,  the  DWT  was  made  shift-invariant  by  omitting 
the  downsampling  process.  This  gives  shift-invariances,  but  it  is  at  the  expense  of 
computing  efficiency. 

In  Principal  Component  Analysis  fusion,  images  to  be  fused  are  expressed  as  column 
vectors  of  their  pixels  values.  The  two  columns  that  results  become  the  two  columns  of 
X.  The  PCA  transform  is  defined  as 


Y  =  A(X  -  mx)  (1.4.3. 3) 

Here  X  contains  the  source  images,  mx  is  the  mean  of  X,  A  is  the  matrix  of  eigenvectors 
of  X,  and  Y  is  the  transformed  data.  The  fused  image  Y  of  the  zero-mean  image  is  then 
added  to  the  pixel  average  of  the  two  source  images.  Here  F(x,y)  is  the  fused  image, 
Fd(x,y)  is  the  output  of  the  PCA  transform,  and  Favg(x,y)  is  the  average  of  the  two  source 
images. 


F(x,y)  =  aFavg(x,y )  +  0Fd(x,y)  (1.4.3. 4) 

Each  of  the  image  fusion  techniques  discussed  above  (with  the  exception  of  Gabor 
fusion),  were  applied  to  LWIR  and  MWIR  images  of  an  urban  scene.  The  original  and 
fused  images  are  shown  below: 
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Figure  1.4.3-1  MWIR  image 


LWIR  image 


Figure  1.4.3-2  Pixel-averaged  image 


DWT  image 


Figure  1.4.3-3  PCA  image 


SIDWT  image 


1.4.3.3  Image  Fusion  Metrics 

Mutual  Information  uses  the  entropy  of  an  image  to  approximate  the  amount  of 
information  in  a  fused  image.  For  a  fused  image/that  results  from  two  input  images  a 
and  b,  the  mutual  infonnation  is  defined  as 

I(a,b,f)  =  H(a,b )  +  H(a  -  b )  - H(a,b,f)  (1.4. 3. 5) 

Where  entropy  is  defined  as: 

H(X)  =  -X^(x,)log2(Pv(x;))  (1.4. 3. 6) 

i 

The  first  two  entropy  tenns  describe  the  input  images  and  increase  the  mutual 
information  value.  The  third  term  considers  information  present  in  the  fused  image  that 
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was  not  present  in  the  source  images  to  be  noise.  For  this  reason  the  third  entropy 
decreases  the  resulting  value  of  the  mutual  information. 

The  other  fusion  metrics  used  here  (fusion  quality  index,  weighted  fusion  quality  index, 
edge-dependent  fusion  quality  index)  are  based  on  the  image  quality  index.  The 
definition  of  the  image  quality  index  is  given  below: 


<2o  = 


4o~,yXy 


(x~  +y  )(crx  +  ctv) 


2  xy  2  axay 

—2  —  2  2  2 

o'xO'y  x  +y 


(1.4. 3. 7) 


Qo  are  found  for  a  moving  window  and  then  averaged  together  to  give  the  quality  of  an 
image  relative  to  another  image.  The  resulting  Qo  term  is  a  value  between  -1  and  1. 

The  fusion  quality  index  compares  a  fused  image  to  its  two  source  images  and  produces 
an  output  between  -1  and  1.  It  is  detennined  from  the  Qo  tenns  and  the  relative  weights, 
X(w),  of  the  input  images.  a(w)  is  a  number  between  0  and  1  that  describes  the  relative 
weighting  of  a  source  image  in  the  fused  image.  The  weight  is  determined  by  the  ratio  of 
an  image’s  saliency  to  the  sum  of  the  saliencies  of  both  of  the  images.  We  used  the 
variance  to  represent  the  salience  of  windowed  sub-image.  Fusion  quality  index  is 
defined  as 


Q{a,b,f)  =  j~T  (a,f  |  w)  +  (\-A(w))Q0(b,f  \  w))  (1.4.3.8) 

weW 


The  weighted  fusion  quality  index  weights  individual  windows  in  the  image  according  to 
the  total  saliency  of  the  image  in  that  window.  This  is  different  from  the  fusion  quality 
index,  which  weights  all  windows  in  an  image  equally.  A  window’s  saliency  is  given  by 

C(w)  =  max(s(<2  |  w),s(b  \  w))  (1.4. 3. 9) 

The  calculation  of  the  weighted  fusion  quality  index  varies  only  slightly  from  that  of  the 
fusion  quality  index 

J _ 

C(W 

w'eW' 

The  intention  of  the  weighted  fusion  quality  index  is  to  give  more  weight  to  areas  of  the 
image  where  more  information  is  present.  If  the  variance  is  a  good  indicator  information 
content,  this  metric  should  yield  more  meaningful  results  than  the  fusion  unweighted 
fusion  quality  index. 


Q(a,b,f)  = 


z 


') 


'ZC(W)(A(w)Q0(a,f  |  w)  +  (\-A(w))Q0(b,f  \  w))  (1.4.3.10) 


The  edge-dependent  fusion  quality  index  is  an  extension  of  the  weighted  fusion  quality 
index.  It  is  the  weighted  fusion  quality  index  of  the  two  input  images  multiplied  by  the 
weighted  fusion  quality  index  of  the  two  extracted  edge  images.  This  is  done  to  model 
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the  importance  edges  in  images  to  human  observers.  The  edges  are  extracted  from  the 
input  images  by  convolution  with  a  high  pass  filter  kernel.  The  equation  for  the  edge- 
dependent  fusion  quality  index  is  given  below  in  terms  of  the  weighted  fusion  quality 
indices. 

(Q,(a,bj)  =  Qw(a,b,f)Qw(a  \b  ’/’)  (1.4.3. 1 1) 

In  the  Mannos-Sakrison  filter  technique  the  fourier  transform  of  the  input  and  fused 
images  are  multiplied  by  a  bandpass  filter  that  describes  the  sensitivity  of  the  human  eye 
to  spatial  frequencies.  The  filter  expressed  in  the  frequency  domain  is 

0(r)  =  2.6(0.0192  +  0. 144r)e~(0 144r)‘  ‘  (1 .4.3. 12) 


In  the  equation  above  r  is  equal  to  the  radius  in  pixels  from  the  center  of  the  shifted  two 
dimensional  fourier  transform.  The  mean  squared  error  of  the  filtered  fused  image  to  the 
each  filtered  input  image  is  found. 

Using  the  different  fusion  metrics  discussed  here,  the  fused  LWIR/MWIR  images  from 
the  urban  scene  shown  previously  were  measured.  The  results  are  shown  in  the  following 
table: 


FQI 

WFQI 

EDFQI 

MINF 

MS(F-L) 

MS(F-M) 

Average 

0.7783 

0.8433 

0.5810 

4.6896 

3257.8 

7037 

DWT 

0.7263 

0.8916 

0.7468 

3.2733 

1218.9 

206.9 

PCA 

0.6011 

0.8710 

0.6842 

3.5897 

1067.9 

173.9 

SIDWT 

0.7854 

0.8723 

0.6870 

3.8848 

656.3 

218.5 

Table  1.4.3-1  Summarizes  the  results  of  the  different  metrics  for  fused  imagery. 

1.4.3.4  Perception  Test  to  Determine  Observer  Preference 

The  results  from  the  different  fusion  metrics  are  clearly  mixed.  Different  measures  of 
image  fusion  give  different  preference  orders.  A  paired  comparison  perception  test  was 
performed  using  a  group  of  human  observers  [Silverstein,  2001].  Two  pairs  of  registered 
LWIR  and  MWIR  images  were  used  as  the  source  images.  For  a  given  scene,  each 
observer  was  shown  pairs  of  images  fused  using  PCA,  pixel  averaging,  Gabor  wavelets, 
discrete  wavelet  transform,  and  the  shift-invariant  discrete  wavelet  transform.  The 
original  LWIR  and  MWIR  images  were  also  used  in  the  comparison  test.  The  observer 
indicated  for  each  image  pair  which  image  was  of  better  quality.  For  each  observer  a 
preference  matrix  was  constructed  from  the  test  results.  In  the  matrix,  tenn  Cy  represents 
the  number  of  times  that  an  observer  preferred  image  i  over  image  j.  The  individual 
preference  matrices  are  all  summed  together  for  analysis.  The  preference  matrix  can  be 
used  to  find  the  percent  of  the  trials  in  which  one  image  is  preferred  over  another.  The 
difference  in  perceived  quality  between  the  two  images,  q,  -  qj,  can  then  be  measured 
between  the  two  images  in  the  following  way: 


22 


(1.4.3.13) 


d[j  =  qt  -q>  ~  V2Z 


C, 


c  +c 

Sv 


In  the  above  equation  Z  is  the  inverse  cumulative-normal  function.  Here  the  assumption 
is  made  that  observer  preference  increases  along  a  cumulative  normal  as  the  image 
quality  increases.  So,  from  a  matrix  of  Cy  values  a  matrix  of  d’y  terms  can  be 
constructed.  Knowing  the  distances  in  quality  between  each  pair  of  images,  all  the 
images  can  be  plotted  along  a  one  dimensional  line  after  one  of  the  images  is  arbitrarily 
assigned  a  q  value  as  a  reference.  A  problem  occurs  if  there  exists  a  pair  of  images  in 
which  one  image  is  always  preferred  over  the  other  across  all  of  the  observers.  In  this 
case,  the  Z  function  returns  infinite  values  and  a  difference  in  quality  cannot  be 
estimated. 


For  this  method  of  analysis  the  following  assumptions  must  be  valid: 

1 .  Each  sample  must  have  a  single  value  that  can  describe  its  quality  q;. 

2.  Each  observer  estimates  the  quality  of  this  sample  with  a  value  from  a  normal 
distribution  around  this  actual  quantity. 

3.  Each  sample  has  the  same  perceptual  variance. 

4.  Each  comparison  is  independent. 

The  results  of  this  test  will  tell  which  fusion  method  observers  feel  provides  them  with 
the  greatest  amount  of  information  with  the  minimum  distortion.  The  quality  values 
found  in  the  perception  experiment  will  be  correlated  with  the  results  of  the  fusion 
metrics  described  previously.  The  metric  which  best  fits  the  output  of  the  paired 
comparison  test  is  likely  to  be  the  most  useful  measurement  tool.  Further  research  may 
include  a  task-performance  perception  experiment  in  which  observers  are  required  to 
detect  or  identify  targets  in  fused  images.  Presently  a  set  of  vehicle  images  suitable  for 
image  fusion  is  not  available.  For  this  reason  the  more  subjective  paired  comparison  test 
was  employed  in  place  of  a  more  objective  detection  or  identification  test. 

The  fusion  effort  will  fonn  another  paper  for  submission  to  a  suitable  journal  once  the 
perception  experiment  results  are  obtained  and  analyzed. 


1.4.4  Planned  Future  Activities 

The  active  sensor  modeling  work  will  be  concluded  by  submitting  a  paper  for  journal 
publication.  Other  than  finishing  the  writing,  the  remaining  work  is  to  use  field  data  to 
validate  the  simulation  and  proposed  simple  model.  Plans  are  to  submit  the  paper  near 
the  end  of  the  first  quarter  of  year  2  activities. 

Perception  experiments  are  planned  for  the  fusion  metric  effort  during  August,  2006. 

The  point  of  the  experiments  is  to  help  quantify  the  meaning  of  the  various  metrics.  That 
is  the  metrics  will  be  compared  to  human  observer  perfonnance  to  calibrate  the  metrics. 
Once  that  data  is  analyzed  and  incorporated,  a  paper  will  be  submitted  for  journal 
publication  probably  during  the  second  quarter  of  year  2  activities. 
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The  image  processing  techniques  modeling  will  continue  during  year  2  of  Center 
funding.  Intelligence/Surveillance/Reconnaissance  (ISR)  sensors  will  be  studied  to 
determine  which  of  the  imager  processing  techniques  are  beneficial  to  those  sensors. 
Performance  modeling  for  those  techniques  will  be  then  be  developed.  As  for  tactical 
sensors,  this  will  involve  perception  experiments  to  validate  analytical  modeling  efforts. 
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1.5  Milestone  -  Performance  Modeling  of  Advanced 
Architecture  Systems  (Dr.  Steven  Griffin) 

Prior  performance  models  are  physics  based  descriptions  of  the  components  and 
subsystems  that  fonn  images  in  the  non-terahertz  regime.  These  are  generally  either 
infrared  (IR)  or  visual  optical  modeling  systems  or  electromagnetic  field  (EMF)  based 
modeling  systems  in  the  millimeter  wave  (rnmwave)  or  at  radio  frequencies  (RF). 
Though  the  terahertz  (THz)  regime  has  many  characteristics  in  common  with  these 
neighboring  regions,  some  aspects  are  significantly  different.  After  extensive  research 
and  a  number  of  meetings,  a  general  approach  to  THz  modeling  was  established.  Prior 
details  were  summarized  in  the  three,  six,  nine,  twelve  and  fifteen  month  reports.  Details 
of  the  completion  of  milestone  one,  two  and  three  were  reported  in  the  progress  reports. 

This  work  compliments  and  enhances  existing  Anny  Research  Laboratory  (ARL) 
programs  by  concentrating  on  selected  issues  of  mutual  interest.  This  includes 
corrections  for  deviations  from  the  classical  optical  performance  model  descriptions.  On 
the  first  year  proposal  cycle,  three  milestones  received  particular  emphasis. 


1 .5.1  Modeling  of  antenna  coupled  FPA  THz  detector  techniques  into 
existing  performance  models. 

Direct  interactions  with  ARL  Maryland  and  NVESD  Virginia  have  established  the 
current  state  of  the  art  with  regard  to  antenna  configuration,  probable  focal  plane  array 
(FPA)  configuration,  and  perfonnance  modeling  issues.  Components  considered 
included  microbolometer  arrays,  heterodyne  sensors  (semiconductor  /  superconductor) 
with  particular  emphasis  placed  on  components  incorporating  arrays  of  antenna  couplers 
versus  those  without.  The  results  identified  significant  differences  and  causative  factors 
which  have  been  used  to  make  perfonnance  modeling  recommendations. 

In  focal  plane  arrays  there  is  a  demonstrated  need  for  antennae  as  coupling  elements. 
This  coupling  accomplishes  several  significant  goals.  It  provides  impedance  match 
between  free  space  and  the  characteristic  impedance  associated  with  the  THz  detectors. 
Given  the  relatively  poor  sensitivity  of  THz  detectors,  system  designers  cannot  afford 
significant  signal  losses  due  to  impedance  mismatch  between  the  optical  image  and  the 
detector  elements.  In  addition  detectors  are  typically  rather  small  compared  to  the 
minimum  sot  sizes  obtainable  by  THz  optics.  Microbolometers,  for  example,  perform 
better  as  their  thennal  conductance  cross  section  is  reduced  driving  designs  to  small, 
beam  lead  elevated  structures.  The  minimum  THz  spot  size  is  typically  larger  than  a 
wavelength  squared.  Thus  only  a  small  percentage  of  the  optically  delivered  power  falls 
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on  the  detector.  An  antenna  is  needed  to  geometrically  couple  the  large  spot  to  the  small 
detector.  As  in  all  focal  plane  arrays,  surface  area  is  required  to  route  interconnects  on 
the  integrated  circuit.  Typically  antennae  are  elevated  on  insulating  material  and  in  order 
to  accomplish  geometrical  coupling  are  larger  than  their  corresponding  detector.  This 
creates  surface  around  the  detector  that  is  available  for  interconnect  routing  without 
loosing  sensitive  area  at  the  antennae  plane. 

Current  antenna  implementations  for  focal  plane  arrays  emphasize  frequency  independent 
and  modifications  of  frequency  independent  planar  antenna  structures.  Planar  is  chosen 
to  ease  implementation  in  the  integrated  circuit  manufacturing  process.  Frequency 
independent  operation  eases  design  and  manufacturing  tolerance  issues  for  theses 
extremely  high  frequencies.  Small  variations  in  material  properties  and  dimensional 
control  are  not  as  likely  to  drive  antenna  frequency  responses  out  of  the  range  of  interest 
or  prevent  them  from  functioning  at  all.  In  a  typical  application,  high  permitivity 
material  is  in  close  proximity  and  corrections  for  high  pennitivity  distortion  to  antenna 
gain  patterns  need  to  be  implemented.  In  addition  the  impact  of  these  deviations  from  the 
lambertian  performance  usually  incorporated  into  optical  theory  need  to  be  considered. 

To  address  the  first  issue,  a  survey  of  current  implementations  was  used  to  detennine 
common  antenna  coupling  configurations  for  THz  focal  plane  arrays.  These  include  2D 
techniques  such  as  bow-tie  or  log  spiral  antennas  placed  above  the  corresponding  THz 
detection  elements  as  well  as  3D  approaches  such  as  semiconductor  micro-lens  elements 
formed  above  the  THz  detection  elements.  Representative  configurations  were  chosen 
from  each  category  including  items  such  as  the  planar  bow-tie  and  micro-lens  which  were 
modeled  using  both  MatLab  and  FemLab  (Comsol).  For  the  purposes  here,  a  bow  tie 
planar  antenna  (see  figure  1.5-1)  over  high  resistance  Si  was  chosen  as  illustrative  of  the 
results.  A  finite  element  analysis  of  the  radiation  pattern  generated  was  generated. 

The  simulations  used  high  resistance  Si  typical  of  integrated  focal  plane  arrays  at  650 
GHz.  Details  of  the  material  performance  is  limited  but  available  from  sources  such  as 
[Bolivar, 2003].  For  this  application  typical  parameters  are: 

e’=l  1.66  with  tan  8  =0.004 

.-.  ec=l  1.66  -0.004664  j  &  cr  =1.9xl010  MKS 

Slightly  less  than  100,000  point,  boundary  condition  driven,  finite  element  analysis  of 
antenna  radiation  problems  was  used  to  simulate  both  the  near  and  far  field.  Some  hand 
adjustment  were  made  to  the  mesh  to  improve  the  element  quality  generated  by  the 
automated  mesh  generators.  The  results  were  post  processed  for  graphical  presentation. 
Classic  free  space  configurations  were  used  to  verify  solutions  and  stability.  Additional 
simulations  were  run  for  configurations  of  interest. 
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(B) 

Figure  1.5-1  Finite  Element  Analysis  of  Common  THz  Coupling  Antenna. 
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Figure  1.5-2  Power  Flux  Density  on  Near  and  Far  Field  Surface  as  Example  of  Non- 
Lambertian  Behavior 
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The  results  are  shown  in  figure  1.5-1  (B)  and  figure  1.5-2.  To  avoid  presentation  of 
reams  of  data  a  pseudocolor  scheme  was  used  in  figure  1.5-1  (B)  where  red  indicates  a 
maximum  power  flux  density  and  blue  near  zero  power  density.  As  can  be  seen  in  figure 
1.5-1  (B),  the  650  GHz  emission  is  modified  by  the  presence  of  the  high  pennittivity 
material.  Several  lobes  are  observed  along  the  x  axis  with  approximate  axial  symmetry 
along  the  y  axis.  The  usual  null  is  observed  at  the  z  axis.  What  is  unusual  is  the 
redistribution  of  the  emission  (and  reception)  energy  to  the  lobes  closest  to  the  Si  surface. 
Though  not  clearly  visible  here,  significant  power  is  also  being  channeled  below  the  Si 
surface  and  out  the  edge  of  the  structure.  For  each  of  the  figures  here  detailed  numeric 
files  can  be  provided  on  request. 

In  figure  1.5-2,  power  density  emission  (reception)  are  plotted  versus  a  polar  angle 
defined  off  the  z  axis  along  the  x  axis.  In  the  near  field  the  pattern  contains  high  angular 
frequency  variations  typical  at  ranges  where  the  details  of  the  antenna  structure  have  not 
smooth  out.  In  the  far  field  pattern,  these  geometrical  details  have  evened  out  producing 
a  typical  far  field  pattern  except  the  power  density  is  redistributed  to  high  polar  angle 
values.  Even  in  the  near  field,  the  bulk  of  the  energy  has  been  diverted  from  normal  to 
the  surface,  to  approximately  parallel  to  the  surface.  This  is  the  usual  tendency  of  high 
dielectric  constant  material  to  concentrate  the  electric  field  close  to  the  substrate  surface, 
distorting  the  usual  antenna  pattern  associated  with  this  antenna  type.  Though  treatments 
have  been  proposed  in  some  applications,  most  focal  plane  arrays  use  architectures  that 
make  no  effort  to  compensate  for  the  distortion.  Even  in  cases  where  compensation  is 
applied,  the  resulting  pattern  is  typically  not  close  to  a  lambertian  pattern  assumed  by 
most  performance  models. 

Since  standard  models  tend  to  assume  a  lambertian  surface  behavior  and  the  antenna 
based  THz  focal  arrays  exhibit  behaviors  far  from  this,  some  compensation  for  the 
antenna’s  selective  filtering  of  the  spatial  frequencies  needs  to  be  incorporated.  A 
number  of  alternatives  were  considered  and  a  performance  modeling  technique  was 
proposed,  demonstrated  and  published.  As  a  summary  consider  the  following. 

In  the  traditional  Fourier  Optical  description  of  an  imaging  system  the  lambertian 
assumption  is  deeply  ingrained  in  the  model  and  strongly  interactive  with  the  paraxial 
assumption  (cos  O  ~  1)  .  In  antenna  based  focal  plane  arrays  the  higher  spatial 
frequencies  traverse  the  edge  of  the  collecting  optics  and  strike  the  detector  at  the  largest 
polar  angle;  thereby,  suffering  the  greatest  relative  attenuation.  This  modifies  the 
effective  MTF  of  the  system.  In  figure  1.5-3  the  marginal  rays  represent  the  higher 
spatial  frequencies  arriving  at  the  detection  element  at  extreme  angles.  The  right  most 
ellipse  represents  the  antenna  reception  pattern  which  has  the  greatest  gain  on  the 
system’s  axis  of  symmetry.  This  gain  attenuates  the  higher  spatial  frequency  information 
arriving  along  the  marginal  rays,  thereby  modifying  the  total  effective  system  MTF  from 
the  normal  description. 
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Figure  1.5-3  Diffraction  theory  modified  for  use  in  THz 


Interfacing  the  antenna  description  into  diffraction  theory  can  be  accomplished  by 
incorporating  the  antenna  power  gain  description  into  the  MTF  description  of  an  imaging 
system. 

Starting  with  a  classic  description:  I  =  Irradiance  Ho  =  Hankel  Transform 

OTF  =  Optical  Transfer  Function 
Ga  =  Antenna  Gain 

For  symmetric,  clear  aperture  [GASKILL,1978]: 
l,{r)  =  H,{H„(l0{r))OTF(p)} 

4f)=  GM  H0{H„(l0(r))  OTF(p)}=Ho{H0(lo(r))[GM  OTF(p)]} 
OTFr  =  Ga{d)  OTF(p) 

To  compute  the  antenna  modified  MTF  -  OTF  and  Ga  need  to  be  related. 

For  symmetric  clear  aperture  MTF  =  |OTF|  =  OTF. 


Insert  into  previous 


G,\e)MTF\p-,pc) 


+  - 


mtfr(p)=Imtfi(p;aJg40)+^ 

Modeling  in  MatLab  with  the  objective  cut  into  63  distinct  annulus  plus  on  axis  cylinder 
yields  the  MTF  curves  in  figure  1.5-4.  Part  (A)  demonstrates  validity  for  a  classic  non¬ 
aberrating,  clear  aperture  system.  Part  (B)  are  examples  of  the  MTF  subcomponents  to 
be  weighted  by  the  antenna  gains  at  the  respective  antenna  polar  angles.  The  resulting 
MTFr  (the  MTF  as  received  by  the  antenna)  is  described  above  and  more  accurately 
describes  the  effective  MTF  of  a  system  incorporating  antennae  at  the  focal  plane.  As 
validation  of  the  model  consider  figure  1.5-5.  In  part  (A)  a  classic  spatial  delta  function 
is  presented  at  the  object  plane  producing  the  anticipated  point  spread  function  at  the 
object  for  an  assumed  lambertian  antenna  reception. 
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Spatial  Frequency  (per  meter) 

(A)  MTF  curves  for  j=2,  j=32  and  j=63 
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Figure  1.5-4  MTF  Optimized  for  Incorporation  of  Antenna  Pattern 


(A)  Object  (B)  Image 

Figure  1.5-5  Antenna  Modified  Image  Simulator  Classic  Test  Case 

Specific  conclusions  for  this  specific  component  of  the  work  includes:  optical  models  do 
require  modification  for  use  at  THz  frequencies  even  though  traditional  tests  of 
assumptions  are  met,  the  antenna  environment  anticipated  in  THz  applications  needs  to 
be  considered  in  adapting  Fourier  optics  -  in  particular  the  interaction  between  the 
antenna  pattern  and  the  image  formation  process,  a  particular  modeling  adaptation  was 
developed  and  successfully  tested. 

In  the  broader  sense,  direct  interactions  with  ARL  and  NVESD  have  established  the 
current  state  of  the  art  with  respect  to  antenna  configurations,  focal  plane  arrays  and 
performance  modeling  issues.  Particular  modifications  for  performance  modeling  have 
been  developed  and  recommended  for  inclusion  in  next  model  revision. 

In  particular,  it  was  decided  that  the  THz  performance  model  should  be  divided  into  a 
modified  radiometric  model  for  use  in  describing  the  signal  and  a  modified  RF  model  for 
use  in  describing  the  temporal  noise.  A  diffraction  modified  radiometric  model  was  used 
to  describe  temporal  behavior  with  the  antenna  modified  Modulation  Transfer  Function 
(MTF)  description  of  spatial  behavior.  Technical  details  were  published  under  separate 
cover.  A  modified  RF  description  was  used  for  temporal  noise  but  was  found  to  be 
limited  by  the  anechoic  chamber  effect  of  atmosphere.  In  this  model,  perfonnance  was 
described  by  traditional  noise  temperature  /  noise  figure  descriptions.  Spatial  noise  is 
dominated  by  atmospheric  interactions  via  amplitude  variations  with  little  phase 
components.  Spatial  noise  treatment  interacted  with  and  led  to  issues  detailed  in 
milestone  three. 

First  milestone  was  completed  August  2005  and  reported  at  six  month  review. 
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1.5.2  Support  performance  modeling  of  image  formation  components 
for  THz  imaging  techniques. 

FEMLAB  (Comsol)  was  obtained  and  implemented.  Initial  geometric  descriptions  of  a 
representative  image  formation  configuration  have  been  entered.  A  traditional  optical 
description  of  the  representative  image  system  has  been  generated.  The  principle  issues 
that  differentiate  the  performance  of  THz  wave  image  formation  (collection)  optics  from 
behavior  typical  of  the  shorter  wavelengths  have  been  designated.  Imaging  at  a  mission 
effective  “stand  off”  distance  (up  to  100  meters)  requires  a  sufficiently  long  focal  length 
to  produce  the  necessarily  small  angular  sub  tense  for  sufficient  resolution  on  target.  The 
field  of  view  of  the  collection  optics  is  necessarily  small.  Factoring  in  practical 
limitations  regarding  weight  and  maximum  linear  dimensions  for  the  applications 
envisioned  dictates  restrictive  collection  optical  diameters. 

Modeling  was  completed  for  a  variety  of  image  formation  configurations  and 
comparisons  made  between  traditional  optical  descriptions  (Matlab)  and  finite  element 
analysis  solutions  to  the  complete  electromagnetic  fields  (EMF)  solutions 
(FemLab/Comsol).  In  particular  consider  figure  1.5-6.  This  is  an  example  of  a  VDI 
22dB  horn  that  is  integrated  into  their  standard  650  GHz  system.  In  the  existing  imagers 
it  is  placed  approximately  1  meter  from  objective  optics.  Traditional  EMF  solutions  were 
extracted  from  [Johnson,  1961].  Traditional  optical  solutions  from  [Born,  1964]. 
FEMLAB  (COMSOL)  finite  element  analysis  solutions  were  developed  for  the  22dB 
hom.  The  model  was  validated  against  the  cited  standard  references  and  hand 
calculations.  Some  manual  adjustments  were  required  to  the  mesh  distribution  to 
eliminate  numerical  simulation  artifacts.  The  numeric  results  are  too  voluminous  for 
inclusion  here.  Selected  pseudocolor  images  are  presented  for  consideration.  Additional 
results  available  upon  request. 

In  figure  1.5-7  part  (A)  the  power  density  is  encoded  with  red  being  maximum  and  blue 
being  minimum.  In  part  (B)  the  same  information  is  presented  many  wavelengths  down 
the  appropriate  waveguide.  The  results  are  consistent  with  the  standard  EMF 
descriptions  previously  cited.  Part  (A)  is  a  representation  of  the  TEn  mode  of  this  horn. 
Part  (B)  is  the  equivalent  for  a  power  detector  mounted  vertically  in  a  waveguide. 
Additional  hand  techniques  were  used  to  validate  the  numerical  solutions. 

Once  model  validation  was  established,  results  like  those  indicated  in  figures  1.5-8  and 
1.5-9  were  generated.  Consider  figure  1.5-8.  Part  (A)  is  a  pseudocolor  representation  of 
the  power  density  emitted  from  a  waveguide  driven  horn.  Maximum  power  densities 
occur  at  the  waveguide  /  horn  transition  and  are  represented  in  red.  Most  of  the  radiated 
signal  is  significantly  weaker  and  is  seen  in  light  blue.  Examination  of  the  actual  numeric 
data  indicate  consistency  with  expected  classical  patterns.  These  are  represented  as  a 
horizontal  cross  section  in  part  (A).  Part  (B)  is  a  contrast  enhanced  version  of  (A)  which 
brings  the  weaker  fringing  fields  into  view.  These  can  be  seen  more  accurately  in  the 
actual  numeric  data.  The  principle  difference  between  the  classical  EMF  description  and 
these  results  are  the  weak  backward  propagating  fringing  fields  due  to  the  edge  currents 
at  the  mouth  of  the  hom.  These  fields  are  usually  eliminated  from  the  classical  solution 
by  assuming  far  field  and  approximating  them  by  zero.  They  would  be  significant  only  if 
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the  objective  optics  /  beam  splitter  of  the  implementation  were  so  close  to  the  horn  as  to 
violate  this  assumption. 

For  example  see  figure  1.5-9.  In  this  case  a  simple  focusing  element  was  placed  at  the 
mouth  of  the  hom.  The  resulting  power  density  is  shown  with  the  same  pseudocolor 
representation  and  is  backed  by  volumes  of  numeric  data.  The  minimum  spot  size  is 
demonstrated  approximately  %  of  the  way  to  the  top  in  part  (A).  Notice  also  the  high 
power  density  intensity  (red)  in  the  waveguide  due  to  the  standing  wave  generated  by  the 
front  surface  reflection  off  the  focusing  element.  In  part  (B)  the  contrast  enhanced  image 
demonstrates  that  the  fringing  fields  are  appropriately  attenuated.  Notice  also  the 
intensity  distribution  near  the  focal  point.  This  appears  to  be  consistent  with  the  classic 
diffraction  limited  description  [Bom,  1964]  using  Lommel  functions. 

Given  the  weakness  of  the  fringing  fields,  it  is  recommended  that  they  not  be  considered 
in  the  model.  However,  advantage  may  be  gained  from  incorporation  of  Lommel 
function  descriptions  into  future  model  revisions.  In  addition,  the  close  proximity  of  the 
objective  optics  to  the  hom  definitely  needs  to  be  addressed.  This  is  partially 
accomplished  by  the  inclusion  of  the  aperture  affect  of  the  objective  optics  in  the 
radiometric  model.  A  second  order  correction  may  eventually  be  required  to  model 
deviations  from  the  ideal  horn  emission  due  to  coupling  between  the  horn  and  mirror. 
This  is  particularly  true  if  subsequent  revisions  to  the  system  design  reduce  the  hom  to 
mirror  distance. 

Second  milestone  completed  October  2005  and  reported  at  six  month  review. 


Figure  1.5-6  Example  of  22dB  Horn 
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(A) 


(B) 

Figure  1.5-7  TEn  Mode  in  Horn  Mouth  Neglecting  Field  Curvature  (A)  and  TEn  Mode 
Deep  in  Waveguide  (B) 


(A) 


(B) 


Figure  1.5-8  Finite  Element  Analysis  of  Horn  Antenna  w/o  Focusing  Element 


37 


(B) 

Figure  1.5-9  Finite  Element  Analysis  of  Horn  Antenna  with  Focusing  Element 
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1.5.3  Incorporate  THz  sensor  path  and  propagation  performance 
modeling. 

Non-DOD  source  THz  models  have  been  implemented.  These  include  the  performance 
impact  of  absorption,  scattering  and  attenuation  from  the  intervening  media  including  air 
and  humidity.  Modifications  to  include  turbulence  have  been  researched.  Specialized 
software  to  model  turbulence  has  been  obtained.  The  research  results  in  this  milestone 
included  a  modeling  review  and  recommendations  leading  to  future  work  and  follow-on 
contracts. 

1.5.3. 1  Non-DOD  source  THz  models  were  implemented  and  compared 
resulting  in  modeling  recommendations 

In  particular,  prior  sub-millimeter  (millimeter)  wave  propagation  models  from  non  DOD 
sources  were  obtained  and  implemented.  Some  are  summarized  in  table  1.5-1  below. 
Though  several  of  the  naming  nomenclatures  address  mmwaves,  the  range  of  validation 
includes  frequency  ranges  to  650  GHz  and  beyond  or  can  easily  be  extended  into  this 
regime.  The  Liebe  model  seemed  most  complete  and  incorporated  much  of  the  other’s 
features.  It  was  originally  in  Fortran  and  could  easily  be  modified  to  accommodate  the 
features  of  the  others.  Also  the  Fortran  is  still  supported  on  the  University  of  Memphis 
High  Perfonnance  Clusters  (HPC)  and  therefore  allows  the  integration  of  the  older  code 
into  a  newer,  more  capable  platfonn.  These  codes  model  the  propagation  of  THz 
frequencies  through  the  prediction  of  scattering,  absorption  and  dispersion  and 
incorporate  common  molecular  species  such  as  diatomic  oxygen  and  water  vapor. 


Author 

Agency 

Document  Title 

Deinnendjian,  D. 

RAND 

Corporation. 

Far  Infrared  and  Submillimeter  Wave 
Attentuation  by  Clouds  and  Rain 

Falcone,  V.F., 
Abreu,  L.W.  and 
i  Shettle,  E.P. 

USAF 

Atmospheric  Attentuation  of  Millimeter  and 
Submillimeter  Waves:  Models  and  Computer 
Code 

Klein,  M.  and 
Gasiewski,  A.J. 

NO  A  A 

Sensitivity  of  Millimeter  and  Sub-millimeter 
Wave  Channels  to  Clear  Air  Temperature  and 
Water  Vapor  Variations 

1  Liebe,  H.J. 

NTIA 

An  Atmospheric  Millimeter  Wave  Propagation 
Model 

Liebe, H.J.  and 
Layton,  D.H 

NTIA 

Millimeter- Wave  Properties  of  the  Atmosphere: 
Laboratory  Studies  and  propagation  Modeling 

Table  1.5-1  Selected  THz  sensor  path  and  propagation  performance  models 

These  results  were  compared  to  the  existing  models  at  NVESD.  Discussions  were  held  at 
NVESD  in  conjunction  with  ARL  Adelphi,  MD  regarding  existing  models  for  sensor 
path  and  performance  modeling.  Commonly  used  models  were  identified  which  included 
the  Liebe  model  enhanced  from  the  sources  above.  Such  models  are  validated  for  the 
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frequency  region  of  interest,  see  table  above,  and  are  effectively  implemented  in  existing 
performance  modeling  systems.  In  a  Spring  2006  visit  to  ARL  at  White  Sands,  the 
performance  of  the  propagation  model  with  regards  to  turbulence  was  discussed  with 
ARL  personnel.  Also  performance  modeling  was  conducted  under  separate  contract  by 
the  University  of  Memphis. 

As  a  result  it  was  possible  to  compare  and  contrast  the  performance  modeling  issues 
considered  most  significant.  In  particular,  traditional  views  of  absorption  and  scattering 
were  revised  to  more  effectively  incorporate  the  dominant  factors  in  the  THz  region. 
Traditional  RF  noise  considerations  were  mitigated  by  the  presence  of  a  low  energy, 
anechoic  chamber  effect  due  to  the  opacity  of  the  atmosphere  at  these  frequencies.  Some 
minor  technical  adjustments  had  to  be  made  to  the  details  of  the  statistical  analysis.  For 
example  the  modification  of  the  statistical  description  of  noise  due  to  the  photon  energy 
being  small  compared  to  the  thermal  energy;  however,  these  corrections  were  generally 
over  whelmed  by  the  atmospheric  opacity  and  low  emission  level.  Current 
implementations  emphasize  coherent  (Super  heterodyne)  detection  in  order  to  improve 
the  sensitivity  of  existing  commercially  available  THz  sensors.  With  the  gain  and  phase 
sensitivity  of  these  approaches,  temporal  noise  due  to  propagation  does  not  appear  to  be  a 
limiting  issue. 

However,  turbulence  represents  a  unique  challenge  for  propagation  related  image 
performance  modeling  which  is  not  addressed  by  propagation  models  such  as  Liebe. 
Liebe’s  model  tracks  longitudinal  (temporal)  effects  of  the  propagation  path  which 
though  related  to  the  affects  of  turbulence  is  tied  to  them  in  a  computational  intensive 
manner.  For  more  effective  implementation,  turbulence  needs  to  be  tied  to  the  lateral 
(spatial  frequency)  aspects  of  image  perfonnance  modeling. 


1.5.3.2  Specialized  Software 

Specialized  software  to  model  turbulence  has  been  obtained.  GUI  driven  commercial 
packages  such  as  FLUENT  were  obtained  and  investigated.  An  example  of  an 
application  is  shown  in  figure  1.5-10,  which  is  a  3D  CAD  description  of  the  University  of 
Memphis  campus.  Such  results  generally  rely  on  Navier-Stokes  style  descriptions  of 
flow  under  boundary  conditions  with  enhancements  for  Kolmogorov  based  turbulence. 
Critical  parameters  such  as  inner  and  outer  scale  or  viscous  dissipation  limits  are 
arbitrarily  set  by  the  programmer.  Though  effective  for  illustrative  purposes,  this  degree 
of  adjustment  limits  effectiveness  for  prediction. 
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(A)  (B) 

Figure  1.5-10  Traditional  Turbulence  Simulation 
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1.5.3.3  Modifications  to  include  turbulence  have  been  researched. 

Traditional  approaches  to  modeling  the  impact  of  turbulence  on  image  fonnation  for  THz 
as  well  as  the  higher  optical  frequencies  emphasize  a  three  step  technique.  The  first  step 
is  to  predict  the  distribution  of  velocity  and  pressure  fields  from  descriptions  of  material 
properties  and  boundary  conditions.  Secondly  to  propagate  an  electromagnetic  signal 
through  the  results  of  step  one  to  predict  the  impact  of  small  optical  property  variations 
on  the  amplitude  and  phase  of  the  radiation.  Finally  this  result  is  integrated  into  the 
description  of  the  image  via  spatial  frequency  representations  or  equivalent.  Each  is 
considered  in  turn. 


Step  one  may  be  approached  through  the  simulation  of  dynamical  systems  [Frish,1995] 
described  by  Navier-Stokes. 


dv 

dt 


+  v  •  V  v  =  -V  p  +  vvv  +  / 


V  •  v  =  0 

plus  I.C.and  B.C. 

Unfortunately  Kolmogorov’s  k-Theorem  regarding  uncomputability  limits  the 
effectiveness  of  this  [Vulpiani,2003].  Various  approaches  involving  assumptions  or 
approximations  have  been  historically  applied  to  this  problem  which  can  be  broadly 
grouped  [Stanisic,1988]  into  four  principle  thoughts  Komolgorov,  Heisenburg, 
Kraichman  and  Hoff.  The  details  are  not  of  principle  interest  here.  All  represent 
significant  insights  and  all  have  acknowledged  limitations. 


The  principle  limitation  from  a  performance  model  perspective  spans  all  of  them  and 
addresses  the  basic  three  step  approach  outlined  above.  It  generates  large  quantities  of 
unnecessary  information  regarding  the  molecular  mechanics  and  EMF  behavior.  What  is 
really  desired  here  is  to  determine  the  affect  of  turbulence  on  the  propagation  of  a  spatial 
frequency  component  of  an  image.  Without  turbulence  the  propagation  is  the  quadratic 
phase  factor.  Deviations  from  it  in  a  turbulent  atmosphere  represent  the  effect  of 
turbulence.  What  is  required  from  a  perfonnance  modeling  standpoint  is  a  description  of 
the  impact  on  a  single  spatial  frequency  of  propagating  through  a  turbulent  atmosphere 
without  engaging  the  three  step  process  above.  Details  of  this  are  being  developed  in  a 
follow-on  effort. 
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1.5.3.4  Conclusion 

A  result  of  the  comparison  and  contrast  of  the  propagation  performance  models  indicates 
a  number  of  issues  of  significance  from  turbulence.  These  were  developed  from  the 
comparison  of  ARL  White  Sands,  ARL  Adelphi  and  NVESD  efforts.  Firstly  the  impact 
of  humidity  variations  across  the  eddies  (turbules)  present  in  boundary  layer  flow  at  the 
surface  is  particularly  effective  at  phase  shifting  and  attenuating  the  THz  frequencies. 
This  introduces  spatial  variations  in  potential  THz  images  due  to  differing  propagation 
paths.  The  spatial  noise  introduced  has  significant  impact  on  the  performance  modeling 
in  the  THz  regime.  Incorporation  of  this  into  a  general  perfonnance  model  is  hindered  by 
the  molecular  mechanistic  orientation  of  prior  modeling.  Prior  descriptions  of  turbulence 
have  emphasized  Poincare,  Navier  Stokes,  Kolmogorov,  Chaotic  Theory,  etc.  These 
approaches  are  oriented  towards  the  development  and  understanding  of  molecular  motion 
as  it  relates  to  classical  dynamical  descriptions.  For  the  purposes  here,  it  is  the 
interaction  of  the  molecular  dynamical  field  with  the  electromagnetic  image  information 
that  maters.  The  classical  description  tends  to  include  a  great  deal  of  detail  that  is 
extraneous  and  is  cast  in  manner  that  is  mathematically  awkward  for  the  calculation  of 
image  performance.  In  consultation  with  ARL  Whites  Sands  and  New  Mexico  State  a 
need  for  a  more  compact  model  for  incorporation  into  the  description  of  the  imaging 
process  was  identified.  This  was  perceived  as  the  most  significant  performance  modeling 
issue  identified  in  completion  of  milestone  three  of  the  year  one  effort.  In  a  follow-on 
effort  (Summer  2006),  model  components  are  being  converted  to  a  high  performance 
cluster  at  the  University  of  Memphis. 

The  third  milestone  was  completed  April  2006. 


1.5.3.5  Future  work 

For  future  development,  emphasis  will  be  given  to  the  modeling  techniques  originally 
developed  by  Heisenberg.  These  emphasized  the  energy  cascade  from  large  eddies  to 
smaller  eddies  at  the  expense  of  mechanical  details  incorporated  into  the  classical 
descriptions.  Such  an  approach  can  be  more  economically  integrated  into  performance 
models  while  preserving  the  essential  elements  for  performance  accuracy.  Such  a  change 
in  modeling  emphasis  should  reduce  computation  requirements  and  result  in  more 
engineering  effectiveness  for  the  predictions.  Follow-on  research  has  been  pursued  with 
the  intention  of  obtaining  separate  funding  to  support  the  further  development  of  this. 
Team  members  have  been  contact  and  agreed  in  principle.  A  preliminary  model  has  been 
developed  to  support  proposal  efforts. 

Publications  and  additional  proposals  have  been  made  and  received  as  a  result  of  this 
contract.  Follow-on  research  has  been  suggested  by  these  results.  New  interactions  with 
ARL  at  White  Sands,  NVESD  at  Ft.  Belvoir,  RTTC  at  Redstone  and  ONR  have  been 
developed.  Dr.  Griffin  would  like  to  thank  the  sponsor  for  this  opportunity. 
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2.0  Vanderbilt  University  -  Program  in  support  of  the 
Center  for  Advanced  Sensors,  Year  1  Annual  Report 

2.1  Administrative 

This  project  supports  the  mission  of  the  CAS  to  develop  a  fundamental 
understanding  of  the  interplay  between  nanotechnology  and  bio-electronics  with 
intelligent  imaging  sensor  architectures  and  an  intelligent  sensor  infrastructure.  Within 
this  multi-university,  multi-year  effort,  Vanderbilt  Unversity  has  the  following  tasks: 

1.  Relate  Natural  Vision  to  Night  Vision  Signal  Processing  -  the  Bio-Optics  of  Vision 
Research  natural  (living  organism)  optical  sensory  representation  by  neural  assemblies, 
recording  isolated  neural  activity  across  a  network  of  cells.  Determine  the  coding  of 
visual  signals  by  cell  populations  and  achieve  a  working  view  of  the  neural  code  to 
achieve  superior  electronic  approaches  to  rapid  AI  night  vision  image  signal  processing 

2.  Develop  Diamond  Bio-electrodes 

Develop  diamond-based,  monolithic  multi-microelectrode  array  for  in  vivo  recording  and 
micro-stimulation 

3.  Develop  Advanced  IR  display 

Research  nanotechnology  capabilities  for  enhancing  imaging  sensor  perfonnance, 
including  advanced  high  definition  infra-red  displays 

The  Vanderbilt  Program  in  support  of  the  Center  for  Advanced  Sensors  is  on 
schedule  and  spend  plan.  This  report  summarizes  and  describes  the  Year  1  progress  for 
the  CAS. 

2.2  Bio-Optics  of  Vision  -  Professor  A. B.  Bonds 
2.2.1  Statement  of  Purpose 

Our  global  goal  is  to  develop  a  clear  understanding  of  how  the  brain  detects  visual 
structure.  This  infonnation  can  then  serve  as  a  substrate  for  biologically-plausible 
models  for  pattern  discrimination  to  achieve  superior  electronic  approaches  to  rapid  AI 
night  vision  image  signal  processing.  Our  fundamental  innovation  is  to  examine  how 
cooperation  between  cells  in  the  visual  cortex  enhances  the  overall  amount  of  information 
that  is  available  for  discrimination.  Traditional  models  rely  mainly  on  the  information 
contained  in  the  firing  rate/output  magnitude  of  individual  cells/visual  filters.  We  have 
found  that  joint  analysis  of  the  firing  patterns  in  populations  of  neurons  adds  information 
(beyond  the  independent  analysis),  supporting  higher  levels  of  discrimination  of  structure 
and  detail.  The  fundamental  mechanism  for  intercellular  cooperation  is  firing  synchrony, 
within  a  10  msec  window  (2.2  -  1).  We  have  therefore  undertaken  a  series  of 
experiments  to  determine  the  specific  visual  conditions  that  encourage  synchrony  and 
whether  the  resulting  synchrony  enhances  visual  discrimination  (provides  emergent 
information)  beyond  that  which  is  available  from  the  firing  rate  (filter  output  magnitude). 

We  have  subdivided  this  effort  into  two  basic  investigations:  (1)  Study  of 
responses  to  analytical  stimuli  (gratings,  noise  patterns)  to  understand  the  fundamental 
spatiotemporal  relationships  that  underly  the  generation  of  synchrony,  and  (2)  study  of 
responses  to  natural  stimuli  to  elaborate  the  specific  visual  structures  that  synchrony  can 
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discriminate.  The  fundamental  model  for  investigation  is  the  primary  visual  cortex  (Area 
17)  of  the  anesthetized  cat,  into  which  we  insert  a  10  x  10  (total  100)  electrode  array. 
This  typically  yields  discriminable  single -unit  activity  from  60-70  cells  simultaneously. 
Synchrony  between  cell  pairs  is  detennined  using  the  Joint  Post  Stimulus  Time 
Histogram  (2.2  -  2).  Synchrony  between  larger  groups  of  cells  is  quantified  using  a 
unique  method  that  we  have  developed  during  the  previous  project  period  (see  below). 

2.2.2  Summary  of  results. 

(1)  Initial  contact  and  discussion.  We  attended  a  project  meeting  at  Ft.  Belvoir 
VA  in  October  2005.  This  led  to  some  discussion  of  interfacing  of  our  work  with  other 
ongoing  efforts.  Dr.  James  Sumner  forwarded  a  copy  of  our  report  to  Rene  dePontbriand, 
Associate  Director  of  Research  ARL-HRED.  He  in  turn  has  referred  us  to  both  Grayson 
Cuqlock-Knopp  and  Bill  Monaco,  both  senior  vision  researchers  at  HRED.  Discussions 
are  ongoing.  (2)  Sensitivity  of  synchrony  to  fundamental  visual  structures.  We  find  that 
synchrony  is  more  sensitive  than  firing  rate  to  the  degree  of  coherence  of  strictly  motion- 
defined  structures.  (3)  Development  of  multicell  coherence  quan  tification  method.  We 
have  developed  and  exhaustively  tested  a  means  of  characterizing  both  the  quality  and 
quantity  of  interactions  in  populations  of  3  or  more  cells.  (4)  Sensitivity  of  synchrony  to 
natural  image  qualities.  We  have  tested  the  relative  sensitivity  of  both  synchrony  and 
firing  rate  to  three  types  of  image  degradation:  Noise,  blur  and  contrast  reversal.  Neural 
synchrony  was  much  more  sensitive  to  changes  in  noise  and  blur  than  was  firing  rate. 
Neither  synchrony  nor  firing  rate  were  significantly  affected  by  contrast  reversal.  (5) 
Selectivity  of  synchrony  within  cell  assemblies  to  natural  images.  We  find  that  synchrony 
within  assemblies  of  up  to  ten  cells  are  highly  selective  to  specific  images  within  a 
sample  of  4500  natural  images.  This  implies  that  synchrony  is  a  reliable  substrate  for 
sparse  coding  of  particular  image  properties.  Early  results  suggest  that  synchrony  is  a 
vector  for  signaling  contour  integration,  and  that  the  selectivity  reflects  specificity  for 
particular  contours. 

Significance.  These  results  collectively  imply  that  while  firing  rate  (filter  output 
magnitude)  conveys  information  on  the  fundamental  structure  of  images,  synchrony 
provides  more  information  on  image  details,  e.g.,  edges  and  contours.  This  suggests 
that  cooperation  between  filtering  elements  is  necessary  for  optimizing 
discrimination  of  fine  image  structure,  especially  in  conditions  of  masking  by  noise 
and  blur. 

Benchmarks,  2006-2007.  1.  Contour  integration  by  synchrony.  We  will 

characterize  the  reliability  of  contour  integration  by  synchrony  and  determine  how  it  is 
affected  by  natural  interference  such  as  noise,  blur  and  contour  occlusion.  2.  Impact  of 
image  qualities  on  synchrony.  We  will  test  how  spatial  phase  and  frequency  information 
and  high-pass  filtering  modify  synchrony  and  firing  rate.  3.  Coordination  of  synchrony 
with  the  association  field.  Association  field  theory  (2.2  -  3,  4)  links  behavioral 
discriminability  with  the  statistics  of  natural  images.  We  will  seek  similar  links  between 
the  statistics  of  natural  contours  and  synchrony  in  the  visual  cortex. 

2.2.3  Specific  results 

1.  Synchrony  dependence  on  motion  coherence  (2.2  -  5).  As  a  first  test  of  the 
involvement  of  motion  coherence  and  synchrony,  we  have  found  that  synchrony  detects 
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structures  generated  purely  by  motion.  We  compared  synchrony  from  drifting  sinusoid 
gratings  and  moving  dot  patterns.  For  dot  stimuli,  10%  of  the  dark  background  was 
covered  by  randomly  distributed  bright  dots.  In  "figure"  regions  consisting  of  periodic 
bars  of  the  same  spatial  frequency  as  the  gratings,  some  of  the  dots  drifted  coherently, 
while  all  remaining  dots  both  inside  and  outside  these  regions  moved  in  random 
directions.  The  figure  regions  themselves  moved  at  the  coherent  drift  rate,  creating  the 
perception  of  a  drifting  grating  pattern.  203  cell  pairs  showed  synchrony  under  drifting 
grating  stimulation,  and  of  these  63  cell  pairs  showed  synchrony  under  random  dot 
stimulation.  For  those  pairs,  a  Student's  t-test  revealed  that  the  synchrony  amplitude  for  a 
random  dot  stimulus  with  100%  coherence  is  lower  than  that  for  the  drifting  grating 
stimulus  (P<0.001)  but  higher  than  that  for  the  dot  stimulus  with  70%  coherence 
(P0.001;  Fig  2.2  -  1). 

4000-1 


Sinusoidal  Random  Dot  Random  Dot 
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Figure  2.2  - 1  Sensitivity  of  synchrony  (left)  to  dot  coherence 


The  average  firing  rate  for  drifting  gratings  was  twice  that  from  random  dot  stimuli 
with  100%  coherence,  but  the  average  firing  rate  for  dot  stimuli  with  100%  coherence  did 
not  differ  significantly  from  those  with  70%  coherence  (P>0.01;  Fig.  2.2  -  1,  bottom). 
Synchrony  detects  temporally  correlated  structure  even  without  clearly  defined  contours, 
but  spatial  integrity  increases  the  synchrony  strength.  Synchrony,  but  not  firing  rate,  is 
sensitive  to  the  coherency  level  of  temporally  correlated  structure. 

2.  Synchrony  in  larger  assemblies.  Cross-correlograms  (2.2  -  6)  and  techniques  such 
as  the  JPSTFI  (2.2  -  2)  cannot  characterize  the  synchrony  between  more  than  two  cells. 
The  cross-correlogram  has  been  extended  to  include  cooperation  among  three  cells  (2.2  - 
7,  8),  but  the  resulting  display  cannot  be  applied  practically  to  larger  groups. 
Gravitational  clustering  (2.2  -  9,  10)  identifies  cells  that  fire  together,  but  results  are 
qualitative  and  still  based  on  pairwise  distance  calculations.  To  quantify  cooperation 
within  multicell  assemblies,  we  have  developed  a  method  that  will  detect  and  quantify  the 
amount  of  correlated  activity  in  a  large  (n>2)  cell  assembly.  A  fixed  integration  window 
is  inappropriate  for  comparisons  across  larger  groups,  so  our  basic  algorithm  is  designed 
to  reflect  the  relevance  of  group  synchrony  to  post-synaptic  neurons  by  modeling  the 
summation  of  postsynaptic  potentials  (PSPs). 

By  convolving  a  point-event  spike  train  with  a  truncated  (10  msec.)  alpha  waveform 
(yielding  a  PSP  train),  we  derive  a  similarity  measure  that  is  conscious  of  time  (Fig  2.2  - 
2).  PSP  trains  for  a  group  of  cells  can  be  summed  and  filtered  to  identify  spike  times  that 
are  coincident  above  a  threshold.  The  Quantity  of  synchrony  is  computed  as  the  ratio  of 
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the  area  under  coincident  waveforms  to  the  total  area  under  all  waveforms  in  the 
assembly.  The  Quality  of  synchrony  is  an  index  computed  on  the  basis  of  the  average 
overlap  of  coincident  waveforms  and  the  length  of  time  a  group  cooperates.  Unlike  other 
approaches,  this  can  be  applied  to  neural  assemblies  of  arbitrary  size,  not  just  pairs.  This 
algorithm  also  selectively  emphasizes  spikes  that  occur  closely  in  time  instead  of  treating 
all  spikes  within  the  integration  time  period  as  equally  important  events.  To  examine 
characteristics  of  group  membership,  we  can  resolve  and  compare  individual 
contributions.  Also,  by  using  shuffle  and  shift  predictor  techniques  (2.2  -  6),  we  can 
resolve  and  separate  sources  of  synchrony,  i.e.  cooperative  activity  stemming  from 
endogenous 


Figure  2.2  -  2  Schematic  of  similarity  measure,  (a)  Spike  trains  from  3  neurons  in  an 

assembly,  (b)  PSP  trains  created  by  convolving  each  spike  train  with  an  alpha 
function,  (c)  Each  PSP  train  is  filtered  to  preserve  only  portions  of  waveforms 
that  are  coincident  above  a  threshold. 


sources,  stimulus  effects,  and  firing  rate-induced  modulation  of  synchrony. 


We  compared  results  from  spike  pairs  using 
this  method  against  the  JPSTH  method  (2.2  -  2)  and 
identified  not  only  the  same  pair  assemblies,  but  also 
others,  most  likely  due  to  the  use  of  temporal 
weighting  (as  opposed  to  fixed  binning).  The 
uncertainties  of  normalization  and  firing  rate 

dependence  are  minimized  due  to  the  very  large 

ratios  seen  between  chance  and  observed 

probabilities  of  synchrony  as  group  sizes  grow  (Fig. 

2.2  -  3).  Observed  synchrony  was  calculated  by 
averaging  raw  scores  for  39  significant  assemblies 
per  size  group. 


Assembly  Size 


Figure  2.2  -  3  Raw  vs.  chance  synchrony  measured  from  subsets  of  39  cells  when  viewing  a 
natural  image. 

Chance  synchrony  was  calculated  by  averaging  shift  predictors  computed  for  200 
randomly  chosen  assemblies  in  each  size  group.  On  average,  significant  assemblies  of  6 
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cells  contain  1500x  more  synchrony  than  that  expected  by  chance  (2.2  -  1 1). 

This  method  is  advantageous  because:  (1)  It  can  be  applied  to  an  arbitrary  number  of 
cells.  (2)  Wavefonn  parameters  like  PSP  amplitude,  integration  time  and  threshold  are 
adjustable.  (3)  The  spike  trains  from  excitatory  and  inhibitory  cells  can  both  be 
represented  (an  IPSP  train  results  from  convolution  with  a  negative  alpha  function).  (4) 
We  can  normalize  synchrony  scores  by  subtracting  the  shift/shuffle  predictor  to  associate 
coordinated  events  with  specific  sources  of  synchrony.  For  instance,  firing  rate  effects 
can  be  removed  and  analysis  can  be  based  on  endogenous  and  stimulus-induced 
synchrony,  or  firing  rate  and  stimulus  effects  can  both  be  removed  and  the  analysis  can 
focus  on  synchrony  from  only  endogenous  sources.  (5)  Members  of  large  groups  can  be 
predicted  from  the  scores  of  smaller  groups,  thus  drastically  reducing  the  number  of 
group  permutations  that  need  to  be  computed.  (6)  No  binning  or  smoothing  techniques 
are  required.  (7)  This  method  is  based  on  biological  principles  and  the  results  are 
physically  meaningful  (i.e.  a  score  of  0.15  means  that  15%  of  all  waveforms  in  a  group 
were  coincident). 

3.  Synchrony  dependence  on  natural  image  manipulation  (2.2  -  12).  One  challenge  in 
analyzing  synchronous  responses  to  natural  stimuli  lies  in  linking  response  changes  to  a 
specific  stimulus  attribute.  To  investigate  the  properties  of  natural  stimuli  that  encourage 
assembly  formation,  we  conducted  differential  measurements.  One  manipulation  was  to 
measure  a  response  to  a  control  image  and  then  modify  the  image  by  noise  addition.  We 
recorded  from  210  groups  of  neurons  (42  groups  of  2-6  cells  each). 

Five  natural  images,  jittered  by  <0.5°  for  two  secs,  were  degraded  in  five  steps 
(Fig.  2.2-  4).  Cooperative  responses  were  compared  to  those  from  the  original  image  and 


Figure  2.2  -  4  Responses  to  five  different  levels  of  noise  degradation.  Both  firing  rate  (FR) 
and  synchrony  are  normalized  to  1,  and  vertical  scale  is  linear.  Colors 
represent  relative  synchrony  of  different  group  sizes  (n=2-6) 

the  difference  represented  the  consequences  of  the  imposed  modification.  For  50%  noise, 
synchrony  decreased  33.8%  for  pairs  and  44.2%  for  6-cell  groups  (p<0.004)  though 
average  firing  rates  (FR)  did  not  change  significantly  (p  >  0.05).  Synchrony  is  only 
destroyed  by  complete  elimination  of  structure.  For  both  noise  addition  and  low-pass 
filtering,  synchrony  is  reduced  much  more  strongly  than  is  firing  rate,  and  synchrony  is 
sensitive  to  the  degree  of  modification,  unlike  firing  rate.  We  found  no  significant 
change  in  either  response  property  for  contrast  reversal,  indicating  that  synchrony,  like 
firing  rate,  shows  non-linear  rectification  in  complex  cells.  From  these  preliminary 
observations  we  hypothesize  that  while  firing  rate  signals  the  fundamental  structure 
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of  the  image,  synchrony  transfers  information  on  fine  detail,  which  is  disrupted 
either  by  noise  or  low-pass  filtering.  We  do  predict  that  synchrony  will  be  less 
disrupted  when  images  move  behind  noise,  since  the  overall  image  structure  is  still 
present.  These  results  would  be  consistent  with  our  earlier  finding  that  synchrony  signals 
fine,  but  not  coarse,  changes  in  orientation  (2.2  -  1).  On  this  basis  we  expect  that 
synchrony  will  be  less  disrupted  by  high-pass  filtering  and  preservation  of  phase 
information,  which  carries  energy  in  detail,  and  that  both  firing  rate  and  synchrony  will 
be  severely  reduced  by  elimination  of  phase  information,  which  defines  the  image 
structure. 

4.  Selectivity  to  natural  images.  To  investigate  the  relative  merits  of  encoding  by  firing 
rate  or  cooperation  within  neural  assemblies  during  natural  vision,  we  compared  the 
selectivity  of  average  firing  rate  and  synchrony  of  cortical  cells  exposed  to  a  sequence  of 
4500  natural  images.  Using  a  10x10  microelectrode  array,  we  recorded  from  71  complex 
cells  in  the  primary  visual  cortex  of  two  paralyzed  and  anesthetized  cats.  We  randomly 
identified  (without  regard  to  spatial  selectivity  or  RF  location  of  members)  350  neural 
assemblies,  ranging  in  size  from  2  to  8  cells  with  50  assemblies  per  size  group.  We  used 
the  fourth  moment  (excess  kurtosis)  of  the  response  distribution  as  a  measure  of 
selectivity  for  each  assembly.  Response  distributions  with  a  large  kurtosis  are  more 
selective  in  that  they  have  a  high  probability  of  a  weak  response  and  a  low  probability  of 
a  strong  response  for  any  single  image  in  the  sequence.  Therefore,  an  assembly  whose 
response  distribution  is  highly  leptokurtic  (excess  kurtosis  >  0)  will  respond  strongly  to 
only  a  few  images  in  the  sequence  and  is  thus  more  selective.  We  measured  the 
synchrony  and  average  firing  rate  response  distributions  for  each  assembly  over  the  set  of 
images  and  found  that  all  distributions  were  leptokurtic,  but  synchrony  response 
distributions  were  highly  leptokurtic  and  kurtosis  increased  nearly  exponentially  with 
assembly  size.  On  the  other  hand,  the  kurtosis  for  average  firing  rate  response 
distributions  stayed  relatively  constant  across  assemblies  of  different  size.  Synchrony 
was  3.9  times  more  selective  than  average  firing  rate  for  pairs  of  cells  and  302.4  times 
more  selective  for  assemblies  of  8  cells.  These  results  suggest  that  synchronous  activity 
among  neural  assemblies  is  more  selective  than  average  firing  rate  in  the  context  of 
natural  images.  Increasing  assembly  size  enhances  the  ability  of  synchronous  coding  to 


Figure  2.2  -  5.  The  kurtosis  (selectivity)  of  synchrony  response  distributions  (a)  increases 
nearly  exponentially  with  assembly  size,  but  (b)  remains  constant  for  firing  rate 
distributions,  (c)  Example  where  the  aggregate  receptive  fields  for  a  given 
assembly  lie  along  a  complex  contour  in  the  image  with  the  best  synchrony 
response,  but  (d)  follow  no  particular  structure  in  the  image  with  the  best 
average  firing  rate  response. 
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discriminate,  but  there  is  no  information  gain  for  average  firing  rate  when  additional  cells 
are  considered.  We  propose  that  higher-order  features  found  in  natural  images  (e.g. 
complex  contours)  are  responsible  for  the  high  selectivity  of  synchrony  compared  to 
average  firing  rate  because  adequate  descriptions  of  high-order  spatial  correlations 
require  the  coordinated  response  of  multiple  cells.  This  process  is  cumulative,  in  that 
more  complex  structures  require  larger  neural  assemblies  for  accurate  description. 

We  also  note  that  there  was  a  tendency  for  the  images  that  generated  the  most 
synchrony  in  a  group  to  have  continuous  contours  along  the  lines  of  the  receptive  field 
preferences  (Fig.  2.2  -  5c,  above).  This  suggests  that  synchronous  assemblies  are 
important  to  the  process  of  contour  integration.  We  will  explore  the  reliability  of  this 
result  in  future  experiments. 
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2.3  Diamond  Microelectrodes  to  Support  the  Bio-Optic  Task 
2.3.1  Summary  Results 

Quantity  and  quality  of  signal  from  the  life  system  source  are  both  essential  to 
data  collection  success,  and  depend  significantly  on  the  construct  of  the  probes.  As 
described  in  the  proposal,  the  diamond-coated  microelectrodes  fabricated  thus  far  are 
intended  to  provide  more  and  better  signal  than  metal  only  needles  of  previous  design. 
Enough  had  been  learned  (2.3  -  1)  to  realize  that  the  bio-system  would  be  significantly 
improved  if:  a)  the  diamond  probes  were  sharper  (to  <500nm  radius)  and  b)  the 
isolation  of  the  sheath  was  better.  The  fabrication  team  developed  means  to  achieve  this 
in  the  first  year,  including  applying  nanodiamond  film  processing  to  the  probes. 
Specifically,  the  tungsten  needle  electrode  substrate  was  coated  with  nanodiamond  film, 
the  tip  sharpness  <1  micron  radius  of  curvature  obtained  and  the  target  impedance  of  ~ 
1M  Ohms  achieved,  see  Figure  2.3  -  1.  Their  electrochemical  superiority  was  clearly 
demonstrated,  Figure  2.3-2.  A  concept  design  for  next  generation  array  of  electrodes 
was  established,  Figure  2.3-3.  Further  effort  on  this  sub  task  awaits  funding  priorities. 


I  00  A) 


0jQ5  0.1  0.15  02  023  0,3  0.35  O.i  0.45  05 

vofcrje  iVi 
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Figure  2.3  -  1  Diamond  Bioprobe 


Figure  2.3  -  2  Probe  Chemsensor  response 
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Design  for  next  generation  array  of  electrodes 


Next  design  -  better  performance 
with  the  bio-system  if: 

a)  the  diamond  probes  are 
sharper  (to  <500nm  radius)  and 

b)  the  isolation  of  the  sheath  is 
better 

Fabrication  applying 
nanodiamond  film  to  the  probes 

Trial  coat  Tungsten  needle 
electrode  substrate  with 
Nanodiamond  film 

-tip  sharpness  <1  micron 
radius  of  curvature 

-  Diamond  deposition  in  plasma 
for  30  seconds  (for  a  very  thin 
film) 

-  Seal  the  electrode  with  pulled 
glass 

- 1.5  M  Ohms  >  Target 
impedance  >  0.5  M  Ohms 


Figure  2.2  -  3  Next  Design  for  Single  Diamond  Neural  Bioprobe 
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2.4  Develop  Advanced  IR  Display 

As  part  of  the  CAS  Program,  Vanderbilt  proposed  to  conduct  research  with  its  diamond 
nanotechnology  capabilities  for  enhancing  imaging  sensor  performance,  including 
advanced  high  definition  infra-red  displays.  Significant  progress  to  plan,  presented 
below,  was  achieved. 

2.4.1  Background 

A  need  exists  for  an  emitter  in  the  infrared  and  related  frequencies  of  the  spectrum  that  is 
efficient,  strong,  responsive,  tunable,  miniaturizable,  electronically  addressable  and  cost 
effective.  Present  technology,  which  often  resorts  to  "hot  wires",  an  approach  more  than 
a  century  old,  is  fragile,  unreliable,  slow  and  expensive. 

Vanderbilt  has  developed  their  diamond  microfabrication  capability  to  achieve  diamond 
thennal  emitters  [Ref.  2.4  -1,2,  3].  The  "glowing"  diamond,  Fig.  2.4  -  1  can  be 
patterned  into  a  variety  of  configurations  and  sizes  to  achieve  robust,  efficient,  cost 
effective,  high-perfonnance,  IR  (infrared)  or  other  frequencies  emission  element  by 
employing  Vanderbilt's  diamond  deposition  technology.  It  is  possible  to  change  the 
frequency  in  the  device  very  rapidly  across  a  wide  range  of  electronically  tunable  options 
and  further  possible  to  combine  these  elements  into  a  pixel  array.  The  necessary 
elements  to  achieve  this  have  been  demonstrated  at  Vanderbilt  including: 

A.  Pattern  emitting  diamond  elements  with  a  monolithic  microelectronic  interconnect. 

B.  Power  behavior  including  emission  from  deep  IR  to  visible. 

C.  Processing  capability  to  achieve  emission  elements  on  silicon  or  ceramic  substrates, 

attached  or  suspended. 
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Current-voltage  evaluation  of  the  diamond  resistors  showed  that  the  resistors  would 
experience,  due  to  negative  thermal  or  voltage  coefficient  of  resistance,  a  region  of 
rapidly  rising  current  flow  with  increasing  voltage  which  can  be  used  with  proper 
controlling  circuitry  to  control  the  resistor  emission  behavior  around  a  "set  point", 
allowing  large  emission  change  with  a  small  change  in  voltage.  From  this  behavior  and 
the  properties  of  diamond  which  include  the  highest  thennal  conductivity,  highest 
binding  energy  and  lowest  reactivity  of  any  material  known  it  is  an  ideal  candidate 

material  to  serve  as  an  emission  source  for  black  body  (a.k.a.  I^R,  Joule  heating) 
radiation. 

The  current  and  power  densities  observed  from  these  Vanderbilt  diamond  power  resistors 
are  much  higher  than  any  other  material.  For  example,  a  sample  sustained  24.6  W.  Its 
cross  sectional  area  was  1.98  x  10  cm  .  This  yields  a  power  density  of  12.4  MW/cm 
compared  to  a  power  density  >  100X  smaller  for  Ruthenium  Oxide,  a  common  power 
resistor  material.  In  rough  comparison,  a  light  bulb  filament  operates  at  ~50  kW  /cm2. 
The  average  power  sustained  by  a  typical  diamond  microresistor  was  approximately  13 
W.  Current  densities  were  observed  as  high  as  ~9  x  104  A/cm2. 

The  technology  transfer/commercial  ramifications  of  this  technology  can  be  seen  from 
this  following  short  list  from  the  many  possible  applications: 

-  A  very  fast  digital  "TV  screen"  in  the  IR  ,  used  for  biomedical  illumination  or  mapping, 
security  monitoring,  and  combat/weather/medical  image  simulation. 

-  The  diamond  emitter  will  be  a  very  unique  illumination  source  for  chemical  analysis. 
Many  analytical  instruments  for  environmental  quality  monitoring  (air,  water,  factory 
effluent,  automotive  standards,  etc.)  for  food  processing,  pharmaceuticals,  biological 
fluids  (blood,  plasma,  etc.),  oil  refining  {anywhere  that  the  substances  involve  organic 
(Carbon-Hydrogen)  materials}  use  an  IR  beam  to  illuminate,  stimulate,  or  otherwise 
analyze  known  and  unknown  samples. 

-  The  printing/copying  process  -  an  array  of  these  IR  elements,  at  the  microsize 
achievable  with  this  technology,  could  significantly  alter  the  cost  and  size  aspects  of  the 
entire  thermal  printing  industry. 
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Visual  Illumination  Under  Load 


Resistors  are  power  loaded  with 
microprobes  and  tested  under 
vacuum  conditions. 

Samples  are  tested  under 
constant  current  loads. 


At  high  current  the  resistors 
experience  enough  resistive 
heating  to  have  visual 
illumination  in  the  form  of  a 
“red  hot”  glow. 


Figure  2.4  -  1  Diamond  Resistors  Glowing  Under  Load 

Cost  considerations:  because  diamond  element  fabrication  is  from  a  simple  CVD  process 
and  from  inexpensive  materials  (methane  and  hydrogen),  the  IR  elements  can  be 
produced  as  cheaply  as  conventional  microelectronic  structures  such  as  chip  resistors. 
Also,  the  ruggedness  of  diamond  emitters  eliminates  the  requirement  of  a  "bulb" 
configuration  sometimes  encountered  in  present  applications. 

2.4.2  Results 

2.4.2. 1  Overview 

The  first  year,  just  completed  and  the  subject  of  this  report,  had  the  objective  of  design  of 
a  diamond  IR  emitting  element  (as  per  Milestone  Three  in  the  proposal  document  -  The 
prototype  diamond  IR  pixel  element  will  be  designed  in  the  first  year.).  Conduct  research 
with  Vanderbilt  nanotechnology  capabilities  for  enhancing  imaging  sensor  performance, 
including  establishing  design  approach  for  advanced  high  definition  infra-red  displays. 

Objectives  for  the  second  year  and  beyond,  are  fabrication,  characterization,  design 
optimization  and  operating  limit  evaluation  of  diamond  pixel  element  IR  array  and 
research  nanotechnology  capabilities  for  diamond  based  IR  pixels,  including  construction 
of  prototype  advanced  high  definition  infra-red  displays  for  enhanced  imaging  sensor 
performance.  In  subsequent  years,  the  design  and  process  will  be  further  developed  and 
characterized,  the  element  performance  evaluated  for  limit  behavior.  Preliminary  design 
for  a  multi  element  array  will  be  achieved  and  fabrication  and  characterization  will  be 
performed  in  year  2  to  3. 
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2.4.2. 2  Project  Performance 

The  process  masks  for  IR  pixel  elements  were  prepared.  Fabrication  of  the  prototypical 
diamond  micro-resistor  pixel  was  conducted  ahead  of  plan.  The  instrumentation  system 
for  their  dynamic  evaluation  was  configured  and  preliminary  testing  performed  ahead  of 
plan. 

Freestanding  diamond  resistors  were  determined  to  be  the  appropriate  configuration  for 
the  project  objectives.  Investigation  of  their  electrothermal  properties  became  a  planned 
action.  A  freestanding  resistor  has  a  high  thermal  resistance  between  the  resistor  and  the 
substrate  due  to  the  cavity  and  leads  to  a  higher  temperature  of  the  resistor  relative  to  the 
substrate  (2.4  -  4-7).  This  temperature  gradient  can  then  be  sensed.  The  suspended 
resistor  architecture  is  thermally  isolated  and  appears  as  a  point  heat-source  (2.4  -  8). 
Thus  the  freestanding  diamond  resistor  we  are  developing  will  be  superior  to  present 
technology  using  resistive  arrays. 

2A.2.4  Fabrication 

Figure  2.4-2  shows  the  composite  mask  of  the  test  resistors,  cavity  and  metal  terminals. 
The  red  is  the  mask  area  for  the  actual  necked  diamond  resistors.  The  green  area  is  for  the 
cavity  mask  designed  for  the  oxide  sacrificial  layer.  The  blue  area  is  the  metal  masks  for 
the  gold  contacts.  Three  different  size  structures  were  built.  The  resistors  were  designed 
to  have  a  ‘necking’  pattern  to  enhance  and  concentrate  the  current  flow  through  the 
resistor  body.  (Scale-the  largest  blue  square  is  0.8mm  on  a  side). 
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Figure  2.4  -  2  Free-standing  diamond  resistor  mask(left),  side  view  (right) 

Freestanding  diamond  resistors  were  successfully  fabricated;  Figure  2.4-3  are  SEM 
photographs  of  the  achieved  diamond  resistors.  Polycrystalline  diamond  films  ~  3 
microns  in  thickness  were  deposited  on  an  oxidized  Si  substrate  by  a  plasma-enhanced 
chemical  vapor  deposition  (PECVD)  process.  Boron  doping  was  from  a  trimethylboron 
(TMB)  gas  source.  A  double  layer  metal  consisting  of  Ti-Au  was  developed  for  masking. 
A  thin  layer  (550  A)  of  titanium  was  evaporated  onto  the  diamond  resistor  followed  by 
3000  A  Au  forming  an  ohmic  contact.  The  TiAu  layer  with  low  resistance  and  high 
melting  point  is  more  suited  to  high  power  and  high  temperature  applications.  The 
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resistor  geometries  were  achieved  in  the  CVD  diamond  layer  by  direct  etching  in  oxygen 
plasma  using  a  patterned  metal  mask 

Three  different  prototype  resistor  structures  were  fabricated,  figure  2.4-4  for  this  study. 
The  resistors  were  “necked”  in  shape,  as  can  be  seen  in  the  figure,  and  ranged  in  length 
from  50  pm  to  200  pm.  The  necking  pattern  is  designed  to  enhance  and  concentrate 
current  flow.  The  structures  were  10pm  to  100  pm  in  width.  Thermally  grown  SiC>2  of  1 
pm  was  the  layer  the  resistors  were  on  and  was  the  layer  removed  locally  to  form  a  cavity 
under  the  diamond  resistor  around  the  neck  area. 


Figure  2.4  -  3  (a)  SEM  of  freestanding  diamond  resistor  (side  view, 4500  x), 
(b)  SEM  of  resistor  neck  area  (350  x) 


Figure  2.4  -  4  Three  Different  IR  Pixel  Resistors  (R) 
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2.4.2A  Testing 

Characterization  of  the  Microstructures;  Current  Pulse  Heating,  the  intent  is  to 
characterize  the  thermal  response/thennal  properties  and  dynamic  response  (e.g. 
sensitivity,  frequency,  and  dynamic  range)  of  the  suspended  diamond  resistor  in 
responses  to  an  impulse  power  load,  see  Figure  2.4  -  5.  A  resistor  will  be  rapidly  heated 
from  a  powerful  current  pulse  generator  while  the  voltage  drop  and  the  current  are 
measured  simultaneously.  The  diamond  resistor  responds  faster  to  a  step  function  in 
heater  current  than  other  materials  (2.4  -  9)  and  this  will  be  quantitatively  measured. 

The  resistors  were  subjected  to  current-pulsed  heating  for  evaluation  and 
characterization.  A  controlled  current  pulse  was  generated  using  a  Magnavolt™  High 
Voltage  DC  Power  Supply/Pulse  Generator.  A  data  acquisition  computer  running 
LabView™  software  regulated  the  frequency  and  duty  cycle  of  the  pulses.  A  low  level 
TTL  pulse  was  generated  by  the  computer  and  used  as  the  control  signal  for  the  pulse 
generator.  In  this  configuration,  the  pulse  generator  could  deliver  a  large  controlled 
output  (0-5 000 V,  0-5  A)  to  the  load  up  to  1kHz.  Various  pulse  magnitudes  and  durations 
were  used.  Due  to  the  small  geometries  of  the  freestanding  resistors,  contact  probes  were 
located  in  close  proximity.  Tests  were  conducted  in  vacuum,  ~  5.8  x  10'5  Torr,  to  reduce 
atmospheric  ionization  and  subsequent  electrical  arcing  between  the  probes.  The  current 
profile  of  a  typical  freestanding  resistor  is  shown  in  Figure  2.4-6  when  the  resistor  was 
subjected  to  a  short  pulse  at  low  current. 
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Figure  2.4  -  5  Suspended  diamond  resistor  in  response  to  an  impulse  power  load 


0.02: 


Figure  2.4  -  6  Resistor  under  low  current  load  (short  pulse) 
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2.4.3  Summary  and  Plans 

The  objectives  of  Year  1  were  exceeded.  The  prototype  resistors  were  designed  and 
fabricated  and  thermoelectric  characterization  begun. 

In  the  coming  year  2,  as  per  the  original  proposal  and  as  governed  by  the  level  of 
funding,  we  will  continue  to  develop  the  IR  pixel  device  with  a  focus  on: 

1.  Quantifiably  measuring  the  T(t)  response  as  a  f(I*V,t)  input  to  the  diamond  resistor 
elements 

2.  Model  thennal  response  to  physical  and  electrical  parameters  of  the  microresistor 
element  and  its  surroundings.  Specifically,  as  regards  resistor  characterization,  because 
we  have  now  successfully  begun  to  characterize  their  electrical  behavior  under  pulse 
heating,  a  method  of  quantifying  the  thermal  luminescence  of  the  resistors  is  needed. 

3.  Refine  fabrication  processes  with  eye  toward  array  construction  for  a  simple  display 
(addressable  elements). 
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3.0  UAH-CAO  High  Speed  Wavefront  Sensor  Final 
Report 

The  goal  of  this  research  effort  was  to  establish  a  testbed  to  analyze  and  advance  a  high 
speed  wavefront  sensor  concept.  The  concept,  based  around  a  pair  of  Fourier  Transform 
Lenses  (FTL’s),  requires  a  high  quality  optical  system  and  the  appropriate  filter 
functions.  In  this  effort,  a  high  quality  FTL  design  was  completed,  utilizing  commercial 
off-the-shelf  optics,  and  the  codes  were  developed  to  model,  design,  analyze  and  produce 
the  necessary  filters.  Additionally,  the  means  for  producing  the  filters  through  two 
methods  were  completed;  amplitude-only  filters  by  simple  printing  techniques,  and 
phase-holographic  filters.  Thus,  the  goal  of  the  effort,  to  develop  the  testbed  upon  which 
these  high-speed  wavefront  sensors  can  be  analyzed  and  advanced,  was  accomplished. 
Additionally,  a  new  method  of  designing  apochromatic  hybrid  lenses  was  derived,  and  an 
approach  for  transitioning  custom  lens  designs  into  COTS  lens  systems  has  been 
successfully  tested. 

3.1  The  High  Speed  Wavefront  Sensing  System  Concept 

3.1.1  Basic  Configuration 

The  high  speed  wavefront  sensor  to  be  advanced  is  based  on  the  physics  of  Fourier 
Optics,  more  specifically,  analogue  optical  computing.  Analogue  optical  computing  is 
based  on  the  inherent  property  of  a  lens  to  produce  a  Fourier  spectrum  at  the  back  focal 
plane  of  the  lens  given  an  input  complex  wavefront[l].  Thus,  given  the  complex  input 
function  s,  the  back  focal  plane  contains  the  Fourier  2-D  Transform  of  the  input,  S  = 
§{s}.  As  shown  in  Fig.  3.1,  by  inserting  a  complex  optical  structure  in  this  plane,  an 
optical  filter  with  a  complex  transmittance  defined  by  the  function  H,  one  can  manipulate 
the  input  object’s  spectrum  for  a  variety  of  results;  discrimination,  tracking,  image 
processing. 


Uifep) 


Uf(u,v) 


U0  =  ${h{u,  v)  ■  UF  (u,v)} 


Figure  3-1.  Basic  layout  of  the  high  speed  wavefront  sensor. 


The  wavefront  which  has  transmitted  through  the  filter  is  now  the  pure  multiplication  of 
the  incident  wavefront,  S,  and  the  filter  function,  H.  Allowing  this  new  wavefront,  H-S, 
to  propagate  through  a  second  Fourier  lens  yields  the  convolution  of  the  original  input,  s, 
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and  the  impulse  response  of  the  filter,  h.  The  careful  definition  of  the  filter  and  the 
geometry  of  the  final  Fourier  Optics  leg  of  the  system  define  how  the  wavefront  errors 
will  be  sensed.  The  properties  of  the  fabricated  filter,  essentially  a  diffraction  grating, 
can  be  further  optimized  to  enhance  sensitivity  to  individual  aspects  of  the  input,  or  to 
provide  multiple  effects  from  a  single  element. 


This  high  speed  wavefront  sensor  system  is  based  on  the  concept  of  an  analogue  optical 
computer  whose  matched  filters  are,  in  a  sense,  Point  Spread  Functions  for  the  wavefront 
errors  we  intend  to  detect.  As  an  example,  we  assume  the  wavefront  entering  the  system 
has  a  pure  defocus  phase  error, 


U(x,y)  =  U0e 


'020 


where  ^020  is  the  magnitude  of  that  error  and  has  units  of  [1 /length].  If  one  of  the  phase 
functions  the  filter  system  is  trying  to  detect  this  power  error,  then  one  could  immediately 
define  the  filter  as  the  complex  conjugate  of  that  phase  function, 


H{$,n)  =  e 


-j^W)wc 


020 


such  that  when  the  two  are  multiplied  at  the  Fourier  plane,  they  will  yield  an  unaberrated 
wavefront  which,  after  passing  through  the  second  Fourier  optics  leg,  will  yield  a 
diffraction  limited  PSF.  By  looking  at  the  peak  intensity  of  the  PSF,  by  placing  a  pinhole 
at  the  PSF  followed  by  a  single  detector  or  by  using  a  detector  array,  one  could  measure 
whether  the  wavefront  is  corrected.  However,  this  particular  detection  scheme  is  quite 
limited.  As  defined,  it  will  only  match  a  single  magnitude  of  power,  and  crudely  indicate 
deviation  from  that  power[2,3]. 


If,  in  addition,  we  introduce  a  linear  phase  shift,  W001,  to  the  defocus  function  and  then 
fabricate  the  filter  such  that  several  diffracted  orders,  n,  are  generated,  we  can  now  place 
several  detector  systems  at  the  output  of  the  second  Fourier  optics  leg,  each  one  tuned  to 
one  of  the  following  filter  functions: 

^  )w020+Wm-x\n 

n 

Each  diffracted  order  will  be  tuned  specifically  to  detecting  the  phase  where  n  is  a 
positive  or  negative  integer.  Thus,  each  channel  of  this  sensor  will  be  optimally  tuned  for 
integer  multiples  of  the  base  defocus  error,  and  the  output  for  the  various  orders  will  be 
physically  separated  at  the  output  of  the  system.  By  appropriately  wiring  the  output  of  all 
the  detectors,  an  analogue  circuit  which  calculates  arbitrary  defocus  with  higher  accuracy 
could  readily  be  designed. 


3.1.2  New  System  Architectures 

Concept  1:  The  Selectable  Wavefront  Sensor  Approach 
The  complexity  of  designing  filters  and  producing  filters  is  driven  by  how  many 
simultaneous  functions  each  filter  must  provide.  Although  we  have  shown  that  a  simple 
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amplitude  filter  which  will  accommodate  several  filter  functions  can  be  produced,  and 
that  more  complicated  grating  functions  can  readily  be  designed,  the  testbed  we  are 
producing  has  the  additional  ability  to  accommodate  multiple  identical  beamlets,  each  of 
which  can  address  an  independent  filter  function  in  the  filter  plane.  This  approach,  which 
is  discussed  fully  in  the  Q5  Report  and  shown  in  Fig.  3.2,  was  verified  through  an  optical 
model  that  was  produced  in  order  to  evaluate  the  required  specifications  for  the  filter 
functions. 


Figure  3-2.  Beamlets  at  the  filter  plane  (left)  and  at  the  detector  after  encountering  the 

array  of  filters.  Note  that  9  beamlets  (separated  into  3  groups  red,  green  amd 
blue)  are  mixed  at  the  detector  plane. 

The  CodeV  model  mimics  the  behavior  of  the  Wavefront  Sensor  System  testbed 
operating  in  the  Page  Oriented  Holographic  Memory  (POHM)  form.  A  single  PSF  is 
injected  into  the  first  FTL  through  a  grating  which  splits  the  converging  beam  into 
multiple  beams  separated  by  angle.  This  kind  of  beam  splitter  is  extremely  common  and 
commercially  available,  often  called  a  fan  out  grating.  Thus,  each  beamlet  will  pass 
through  the  first  FTL  and  intersect  the  filter  plane  at  physically  separate  locations.  Now, 
at  each  location,  a  different  filter  can  be  encoded  with  a  different  linear  shift  to  alter  the 
location  of  the  detection  spots.  Each  identical  beamlet  is  incident  on  a  different  region  of 
the  filter  plane  and,  hence,  a  different  filter  function.  Each  filtered  beamlet  is  directed  to 
a  different  location  on  the  detection  plane.  The  advantage  of  this  new  configuration  is 
that  each  filter  function  can  now  be  produced  with  a  spatial  light  modulator  (SLM)  which 
allows  for  active  selection  of  what  aberration  functions  are  being  detected.  One  can  leave 
a  few  filter  functions  fixed,  and  vary  the  rest  to  detect  either  specific  magnitudes  of 
aberration  functions  or  vary  the  type  of  aberration  function  being  detected. 

Although  we  have  not  implemented  this  approach  yet,  we  believe  we  have  an  active 
reflective  SLM  in  the  lab  which  we  hope  to  use  in  conjunction  with  a  set  of  fixed  filters. 
Ultimately,  we  intend  to  evaluate  such  a  system  and  publish  our  results. 

Concept  2:  The  Non-discreet  Wavefront  Sensor  Approach 

A  completely  different  approach  utilizes  a  single  filter  at  a  single  order  to  determine 
arbitrary  amounts  of  aberration,  such  as  de focus.  This  approach  requires  a  coordinate 
transformation  on  the  incident  wavefront  which,  for  example,  remaps  the  field  from  an 
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(x,  y)  Cartesian  mapping  to  a  (p,  0)  Cartesian  mapping.  In  addition,  the  (p,  0)  mapping 
can  be  modified  to  be  a  (p',  0)  mapping  such  that  the  phase  function  redistribution  yields 
a  linear  phase  shift  in  1-D.  Thus,  the  filter  function  is  now  given  by  the  equation 

?  IT 

/  \  ~j  ,  ^'^020'^MAX 

H  (4,tj)  =  e  A 

where  we  maintain  the  coordinate  notation  in  the  filter  plane.  We  now  optically  produce 
an  identical  coordinate  transformation  on  the  input  wavefront  and  send  this  modified 
wavefront  into  the  wavefront  sensor.  At  the  filter  plane,  the  multiplication  of  the  incident 
field  and  the  filter  function  yields 

/  \  /020~‘%2o) 

u  2(Z,T})  =  e  A 

which  means  that  the  magnitude  of  defocus  can  be  read  out  at  the  end  of  the  system  on  a 
single  position  sensing  detector  (PSD): 

l(u,  v)  cc  F{U2 }  cc  PSF0  {u  +  %MAX  •  {W020  -7ffm  )A-F2,v) 

where  F2  is  the  focal  length  of  the  second  Fourier  optics  leg.  So,  the  filter  function  now 
detects  an  arbitrary  amount  of  focus  error  by  producing  a  linearly  dependent  voltage 
variation  from  a  single  detector. 

The  challenges  increase  as  one  increases  the  number  of  aberration  types  to  detect.  One 
either  desires  filters  whose  channels  produce  results  that  are  immune  to  confounding 
inputs,  or  a  system  whose  circuitry  can  extract  the  correct  output.  We  have  begun  to 
examine  this  system’s  level  question  with  the  MatLAB  tools  we  have  developed,  as 
discussed  in  the  Q4  Report.  Preliminary  results  are  indicating  that  the  orthonormality  of 
input  functions  is  maintained  after  coordinate  transformations. 

Additionally,  we  have  already  begun  to  process  of  designing  the  coordinate 
transformation  optics  using  diffractive  surfaces.  Bryngdahl[4]  published  some  of  the 
seminal  work  in  this  area.  Two  of  the  researchers  here  at  UAH  have  had  previous 
experience  performing  similar  analyses  and  some  preliminary  designs  have  been 
completed. 


3.2  The  FTL  Optics 
3.2.1  Design  Theory 

The  basic  foundation  of  the  High  Speed  Wavefront  Sensor  is  the  optical  decomposition 
of  the  wavefront  via  Fourier  Transform.  Thus,  the  design  of  the  optical  system  which 
produces  the  Fourier  Transform  is  paramount.  As  such,  the  underlying  theory  for  the 
optical  design  had  to  be  well  understood  in  order  to  ensure  that  the  critical  parameters 
were  accounted  for,  but  that  negligible  parameters  did  not  affect  the  effort.  Thus,  we 
applied  a  simple  yet  powerful  technique  for  laying  out  and  evaluating  optical  systems,  the 
Y-Ybar  diagram[5].  Once  this  framework  was  developed,  a  method  of  chromatic 
aberration  correction  for  this  complex  system  was  applied  in  a  new  manner,  yielding 
some  powerful  approaches  for  optical  system  achromatization  with  hybrid  refractive- 
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diffractive  lenses.  Finally,  a  monochromatic  design  study  began,  initially  yielding  an  all 
custom-optic  lens  design  which  was  successfully  transitioned  to  a  significantly  less 
expensive  all  commercial  off-the-shelf  design.  The  final  design  was  determined  to  have 
relatively  loose  tolerances  and,  as  it  turns  out,  to  be  adaptable  to  a  broad  range  of 
wavelengths  and  still  diffraction  limited. 


3.2. 1.1  Paraxial  methods  -  the  Y-Ybar  Diagram 

The  design  of  an  FTL  must  have  sufficient  aberration  correction  to  both  bring  plane 
waves  to  diffraction  limited  focus  and  the  convert  point  sources  into  diffraction  limited 
plane  waves.  As  discussed  fully  in  the  Q1  and  Q2  Reports,  the  optical  system  must 
therefore  be  well  behaved  at  two  conjugates  simultaneously,  and  these  two  conjugates 
could  not  be  more  different;  object  at  infinity  versus  image  at  infinity.  By  laying  out  the 
system  paraxially,  it  is  apparent  that  it  must  have  neither  image  aberrations  nor  pupil 
aberrations.  Fortunately,  a  simple  approach  exists  to  attack  both  of  these  design 
requirements;  make  the  FTL  symmetric  about  its  center.  In  this  way,  designing  the  lens 
to  have  no  image  aberrations  simultaneously  corrects  pupil  aberrations. 

Another  researcher  at  UAH,  Dr.  Lloyd  Hillman,  had  developed  a  new  linear  algebra 
approach  to  evaluating  image  and  pupil  aberrations,  and  exploring  how  they  are 
connected,  through  rotations  of  traces  on  the  Y-Ybar  diagram.  Dr.  Hillman  passed  away 
in  May  of  2005  before  he  was  able  to  publish  this  work.  Fortunately,  a  student  was 
working  on  a  Masters  degree  looking  at  higher  order  aberration  terms,  and  much  of  the 
work  was  documented  after  Dr.  Hillman’s  death  in  the  students’  thesis.  Several 
researchers  here  at  UAH  intend  to  publish  this  work,  and  the  application  of  his  theories  is 
extraordinarily  well  suited  to  the  design  of  FTL’ s.  The  aberrations  of  the  whole  FTL  is 
identical  to  the  aberrations  of  one  half  of  the  design  plus  an  identical  lens  rotated  90 
degrees  about  the  origin.  This  approach  shows  great  promise  for  defining  what 
aberration  terms  are  critical  at  which  portion  of  the  optical  train,  thus  guiding  the  design 
to  its  optimum  configuration. 


3.2. 1.2  Addressing  Chromatic  Variations 

Initially,  the  optical  design  effort  began  by  considering  a  broad  waveband  system. 
Defining  what  signifies  achromaticity  for  this  system  coupled  to  the  fact  that  the  filters 
were  initially  going  to  be  diffractive,  it  was  anticipated  that  designing  the  appropriate 
system  would  require  a  hybrid  diffractive-refractive  lens  system.  Thus,  a  new  application 
of  a  powerful  color-correction  approach,  the  Buchdahl  Chromatic  Coordinate^, 7],  was 
derived  and  applied,  as  discussed  in  the  Ql,  and  Q2  Reports.  The  Buchdahl  Chromatic 
Coordinate  has  been  successfully  used  to  design  achromats  and  apochromats  from 
standard  available  glasses  and  liquids.  The  extension  of  the  approach  to  diffractive 
systems  has  not  been  employed  until  now. 

Buchdahl’s  chromatic  coordinate  defines  the  dispersion  of  a  material  by  first  making  the 
transfonnation  from  wavelength,  X,  to  the  unitless  variable  (MX), 
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co(A)  = 


A  —  A0 

1  +  a  ■  (A  -  A0 )  ’ 


(3.5) 


where  A  is  given  in  micrometers,  Ao  is  the  center  wavelength,  and  a  is  an  experimentally 
determined  value  which,  for  A0=0.574um,  a=2.5.  Following  this  transfonnation,  the 
refractive  index  of  a  material  can  be  represented  by  the  equation 

n(co)  =  N0  +  v1  -co+  v2  -co2  +  v3  -<z>3h — \-vt  ■  a>‘ .  (3.6) 

Thus,  the  refractive  index  is  now  a  simple  polynomial  expression  with  the  index  at  the 
center  wavelength  equal  to  N0,  and  the  ith  order  dispersion  coefficient  equal  to  v,  in 
Buchdahl’s  coordinate  space.  The  simplification  afforded  by  the  BuchdahTs  chromatic 
coordinate  is  that  the  refractive  index  of  a  material  can  be  well  represented  by  a  low- 
order  polynomial  function;  typically  the  second  order  form  is  sufficient  for  finding  new 
apochromatic  lens  forms. 


The  dispersion  of  a  diffractive  lens  is  purely  a  function  of  the  grating  law,  thus,  its  power 
is  directly  proportional  to  the  wavelength, 

(3.7) 

A0 

By  solving  Eq.  3.5  for  the  wavelength,  A,  and  combining  it  with  the  equation  for  the 
power  of  a  diffractive  lens  yields  the  relation 
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Expanding  this  out  in  a  Taylor  series  of  order  /  about  co  =  co(A0 )  =  0  yields 
(j){(0)  =  0O 
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By  equating  the  bracketed  portion  of  this  equation  to  the  Buchdahl  equation  of  refractive 
index,  Eq.  3.6,  the  coefficients  of  the  dispersion  can  be  computed  as 


a 


v:  = 


4 o 


(3.10) 


and 

N0  =  l.  (3.11) 

Thus,  vl  =  1.74,  v2  =  4.36,  and  v3  =  10.9,  values  which  are  approximately  2,  3,  and  4 
orders  of  magnitude  larger  than  the  respective  coefficients  calculated  for  typical 
refractive  materials. 


Using  these  terms,  one  can  find  the  power  distribution  for  a  refractive  -  diffractive  hybrid 
achromat.  Following  Robb,  et.  ah,  the  power  (linear  approximation)  of  the  refractive  and 
diffractive  elements  are 


<pUo)= 


v, 
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<P,M>  and  (pdiff{A 0): 
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(3.12) 


The  above  equations  can  be  applied  to  finding  the  power  distribution  for  a  hybrid  lens 
with  the  refractive  portion  made  of  the  glass  SF9. 
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7,  =—^=  °- 10603  = -016167  ^r1  and,  N0  =1.655847 

1  iV0  —  1  0.655847  0 

For  the  diffractive  lens,  the  standard  method  requires  the  Abbe  number  and  the  new 
approach  requires  the  first  order  BCC  coefficient.  We  used  wavelengths  F  (486.  lnm),  d 
(587. 6nm),  and  C  (656. 3nm). 


v  = 

V  diff 


587.6 


486.1-656.3 


=  -3.4534; 
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We  define  an  achromat  of  total  power  cpt  =1.0  . 


1)  Standard  method: 
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2)  Buchdahl’s  coordinate: 

V \diff 
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-<p,  =  0.087 . 


Thus,  the  solutions  using  the  standard  approach,  Abbe  number,  and  the  Buchdahl 
Chromatic  Coordinate  yield  comparable  results  whose  differences  are  expected,  given  the 
process  each  uses.  One  finds  that  the  residual  axial  power  is  comparable  for  the  two 
solutions,  the  primary  difference  being  the  location  of  the  crossings.  Note  that  the  focus 
shift  for  comparably  powered  pure  refractive  and  pure  diffractive  solutions  is  0.028mm 
and  0.3mm,  respectively,  versus  the  hybrid’s  0.003mm. 


Where  the  power  of  the  approach  becomes  apparent  is  when  the  number  of  refractive 
elements  is  increased  to  2,  and  the  number  of  chromatic  focal  crossings  is  increased  to  4. 
For  thin  lenses  in  contact,  a  method  to  both  select  the  best  two  glasses  and  to  set  the 
power  distribution  falls  out  of  the  derivation.  This  method  has  been  documented,  along 
with  the  Buchdahl  Coordinate  coefficients  for  a  diffractive  lens  in  a  paper  which  is  being 
re-edited  for  submission  to  Optical  Engineering,  and  in  the  Q2  Report.  In  addition,  a 
variation  of  this  visible  waveband  analysis,  applied  to  refractive-diffractive  hybrid  lenses 
operating  in  the  IR,  was  presented  at  the  SPIE  DSS  conference  in  Orlando,  Session  6206 
as  “Applying  the  Buchdahl  dispersion  model  to  infrared  hybrid  refractive-diffractive 
achromats,”  by  Pi,  Y.  and  Reardon,  P.J  . 


3.2.2  The  Final  Design 

A  number  of  preliminary  diffraction  limited  designs  were  developed  which  were  then 
exercised  to  evaluate  the  aberration  space  around  the  design  solutions.  Both  the  entrance 
pupil  diameter  (EPD)  and  the  field-of-view  were  expanded,  individually  and  in  tandem, 
to  see  what  form  the  lenses  would  be  driven  to,  which  is  discussed  more  fully  in  the  Q1 
Report.  It  was  found  that  increasing  the  EPD  lead  to  significant  amounts  of  higher-order 
spherical  aberration,  but  it  was  compensated  by  the  lower-order  spherical  aberration  and 
slight  defocus.  This  design  would  have  significantly  tighter  tolerances.  When  increasing 
the  field,  however,  the  design  was  primarily  limited  by  Petzval  curvature,  which  is 
inherent  in  the  form.  It  was  accommodated  mostly  by  defocusing  the  system  which 
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means  that,  although  still  diffraction  limited,  the  image  quality  is  reduced  at  the  center 
and  edge  of  the  field.  The  tolerances  of  this  design  would  not  be  significantly  tighter 
than  the  base  design. 

The  initial  optical  design  for  the  final  FTL  system  was  for  an  EFL=  1000mm,  f/10 
telecentric  system  which  was  presented  at  the  2nd  Quarter  Review  in  October.  It 
consisted  of  6  custom  singlets  for  a  single  FTL.  Quotes  to  fabricate  the  lenses  had  been 
received;  $  16k  for  the  lenses  for  the  two  systems.  We  indicated  that  a  COTS  design  was 
being  pursued  to  reduce  costs.  The  new  design,  shown  in  Fig.  3.3,  was  completed  and 
discussed  in  the  Q3  Report.  The  EFL  was  reduced  to  EFL=800mm,  but  the  telecentricity 
and  f/10  were  retained.  The  components,  all  commercial  optics,  cost  -$4400  for  the  two 
systems,  a  major  reduction  in  price  with  no  loss  in  performance  and  only  minor  reduction 
in  aperture  diameter. 

Although  this  system  was  designed  for  632nm  wavelength,  the  design  can  be  operated  as 
diffraction  limited  at  5 14nm  and  532nm  with  only  minor  adjustment  of  airspaces. 


Figure  3-3.  Y-Ybar  diagram  and  layout  of  FTL 

As  the  broad  wavelength  adjustability  indicates,  the  optical  tolerances  for  the  system  are 
extremely  loose.  For  example,  limn  lens  thickness  errors  and  limn  lens  separation 
errors,  randomly  applied,  combine  to  produce  a  system  which  still  is  diffraction  limited, 
but  has  a  less  than  1%  change  in  EFL  and  a  slightly  different  focal  plane  location.  The 
radii  of  the  lenses  were  individually  measured,  so  the  design  with  the  actual  lens  radii 
were  entered  into  the  design  code  and  slightly  reoptimized.  Thus,  even  though  the 
sensitivity  to  radius  errors  is  similarly  low,  we  have  very  accurate  values,  essentially 
eliminating  any  radius  error  influence. 

This  design  is  phenomenal  given  its  cost  and  performance.  In  fact,  the  Space  Bandwidth 
Product  for  this  FTL  system  is  larger  than  for  any  FTL  system  our  team  has  encountered 
(over  10A8).  We  intend  to  publish  this  design  in  Optical  Engineering. 
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Additionally,  the  COTS  lens  transition  approach  utilized  in  this  effort  also  yielded  an 
extremely  cost  effective  and  high  quality  monochromatic  doublet  lens.  The  pairing  of  an 
equi-convex  and  a  plano-concave  lens  can  yield  a  near-zero  spherical  aberration  lens  with 
little  coma.  Thus,  the  following  two  lenses  produce  a  monochromatic  doublet  of 
EFL=770mm  and  EPD=77mm  lens  with  a  20mm  diameter  diffraction  limited  image 
surface,  or  a  10mm  diameter  diffraction  limited  image  at  EPD=95mm.  Comparable 
achromats  are  the  Edmund  Scientific  30976  and  the  Newport  PAC097  which  also  operate 
at  f/10,  but  cost  -$500  and  -$300  respectively,  and  are  limited  to  3”  diameter  collecting 
apertures.  The  Linos  322317  has  a  150mm  diameter,  with  great  performance,  but  costs 
-$3600.  For  comparison,  the  two  lenses  we  specify  in  this  design  study,  Fig.  4.3,  Ross 
Optical’s  L-BCX276  and  L-PCC107,  costs  only  -$180  in  total. 


Figure  3-4.  Perfomance  of  COTS  transitioned  doublet. 


3. 3  Filter  Implem  en  ta  tion 

A  significant  effort  was  placed  on  developing  the  codes  to  define,  model,  print  and  test 
the  filter  sets,  as  is  discussed  in  the  Q4  Report.  After  some  efforts  to  debug  the  MatLAB 
algorithms,  filters  were  printed  using  a  standard  laser  printer,  and  then  optically  tested 
interferometrically  and  by  evaluating  the  PSF’s  of  the  filters.  The  code  was  verified,  and 
the  filters’  theoretical  performance  in  Wavefront  Sensor  System  was  modeled  using 
additional  MatLAB  codes  developed  under  this  effort.  Testing  verified  the  codes  and 
revealed  the  prints’  limitation,  but  two  simple  workarounds  were  conceived. 


3.3.1  Design  Coding  in  MatLAB 

The  required  MatLAB  code  was  developed  to  numerically  model  the  filters.  The  code 
was  initially  directed  to  model  the  Zemike  Phase  functions  for  Z4  (defocus)  and  Z7/Z8 
(horizontal  and  vertical  coma).  The  first  step  was  to  produce  the  general  codes  for 
producing  these  phase  functions  which  was  successfully  completed.  This  was  verified  by 
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quantitative  analysis  of  the  data  generated  in  the  data  tables.  As  the  Wavefront  Sensing 
system  is  based  around  a  Fourier  Transform  lens  system,  coding  the  Fourier  system  was 
done,  using  a  relatively  straight  forward  DFT  approach,  however,  the  code  was  extended 
to  allow  for  Fresnel  or  Fractional  Fourier  transforms.  Thus,  the  “diffraction  pattern”  or 
PSF  of  a  lens  with  the  previously  coded  filter  function  could  be  computed  an  analyzed. 
The  familiar  diffraction  patterns  for  defocus  and  coma  resulted.  Thus,  initially,  two  code 
modules  were  completed. 

One  implementation  for  the  system  is  to  utilize  binary  phase-only  filters.  These  filters 
primarily  produce  two  diffracted  orders,  +/-1,  with  -40%  efficiency  for  each  order.  The 
code  had  to  be  modified  to  accurately  represent  this  new  configuration;  the  continuous 
phase  profile  is  replaced  by  steps  of  0  or  n  radians.  This  unfortunately  now  places  both 
diffracted  orders’  angular  spectrums  to  be  essentially  parallel,  thus  their  correlations  will 
end  up  directly  on  top  of  each  other  at  the  detection  end  of  the  system.  Binary  tilt,  or 
linear  phase  shift,  was  applied  to  the  functions  to  separate  the  orders.  Thus,  if  the 
encoded  phase  filter  is  a  single  wave  of  Z7,  the  filter  should  produce  two  outputs, 
laterally  separated  in  the  detection  plane;  one  for  +Z7  and  the  other  for  -Z7. 


To  model  the  output  of  the  Wavefront  sensor  with  the  Z7  detecting  filter  in  place,  the 
model  assumed  a  PSF  entering  into  the  system,  being  Fourier  transfonned,  and  then 
passing  through  the  filter.  If  a  positive  Z7  is  incident,  the  PSF  has  a  peak  intensity  ~6 
times  that  of  its  complement.  Likewise,  if  the  incident  beam  has  negative  Z7,  the 
negative  Z7  sensing  PSF  has  a  peak  intensity  ~6  times  that  of  its  complement.  If  on  the 
other  hand,  the  incident  wavefront  has  pure  Z4,  the  output  shows  no  distinction  between 
the  two  spots,  hence,  no  indication  of  any  Z7.  Thus,  the  system  will  selectively  indicate 
the  presence  and  amount  of  a  specific  aberration,  within  a  range  defined  by  twice  the 
magnitude  of  the  encoded  of  aberrations. 


Figure  3-5  From  left  to  right,  a  filter's  impulse  response,  the  filter,  and  the  response  of  the 
filter  to  an  input  field  with  +Z7. 


In  the  real  wavefront  sensing  system,  one  can  use  multiplexed  filters  to  detect  several 
aberrations  simultaneously.  To  model  this,  the  aberrations  for  Z4  and  Z7  were  combined 
into  a  single  filter,  as  shown  in  Fig.  3.4,  but  the  linear  phase  shift  was  defined  to  be 
orthogonal  for  the  two  cases.  When  a  Z4  wavefront  error  is  input  into  the  system,  the 
output  clearly  indicates  its  presence  and  shows  no  indication  of  the  orthogonal  Z7  error, 
as  expected.  Similarly,  when  a  Z7  wavefront  error  is  input  into  the  system,  the  output 
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clearly  indicates  its  presence  and  shows  no  indication  of  the  orthogonal  Z4  error.  Thus, 
models  of  the  system  clearly  indicate  that  the  basic  Wavefront  Sensing  concept  is 
feasible. 

As  the  primary  goal  of  this  first  Years’  funding  is  to  develop  a  test  bed  and  software  tools 
to  evaluate  new  filtering  concepts,  a  simple,  rapid  and  inexpensive  way  to  produce  the 
filter  sets  is  desired.  With  the  prevalence  of  fairly  high-resolution  printers,  phase 
functions  can  be  readily  and  accurately  produced  as  amplitude-only  filters.  Advantages 
of  this  approach  include  the  fact  that  the  filters  will,  in  fact  be  inherently  multi-order,  thus 
each  filter  will  produce  spots  with  several  magnitudes  of  the  aberration  to  be  detected. 
This  can  be  utilized  to  either  expand  the  dynamic  range  or  the  resolution  within  a  fixed 
range  of  the  Wavefront  Sensor.  Other  advantages  are  very  fast  production  and  very  low 
price.  The  primary  disadvantage  is  the  relative  low  efficiency  as  a  significant  portion  of 
the  beam  transmits  undiffracted,  and,  depending  on  the  duty  cycle  of  the  pattern,  the  first 
orders  terms  range  from  10%  to  25%.  A  MatLAB  routine  was  written  to  correctly  design 
and  model  these  amplitude-only  filters. 


3.3.2  Filter  Testing 

Filter  testing  was  critical  to  verify  that  the  code  was,  in  fact  producing  the  correct  phase 
functions  both  in  scale  and  form.  Two  approaches  were  utilized;  the  first  was  to  examine 
the  focused  spot  after  a  beam  propagates  through  the  filter  (the  impulse  response  of  the 
filters),  and  the  second  was  to  directly  measure  the  phase  with  an  interferometer. 

To  directly  measure  the  PSF  of  a  lens  system  having  first  passed  through  the  filter,  the 
filters  were  mounted  in  front  of  a  collimated  HeNe  source.  For  this  purpose,  a  Zygo  PTI 
interferometer  producing  a  4”  diameter  beam  was  used.  The  beam  was  then  focused  by  a 
large  diameter  focusing  lens,  passed  through  an  attenuator  and  onto  a  Pulnix  TM-9701 
progressive  scan  camera  at  the  focus  of  the  lens.  The  data  was  grabbed  through  a 
National  Instruments  PCI  1422  frame  grabber  card  using  Labview.The  data  showed  that 
the  filters  were,  in  fact,  inducing  the  desired  phase  errors  onto  the  beam,  but  there  was 
difficulty  separating  the  orders  due  to  the  relative  short  focal  length  of  the  focusing  lens, 
and,  therefore,  difficulty  in  seeing  many  of  the  orders  at  the  same  time.  Additionally,  the 
scale  of  the  PSF’s  were  only  clearly  resolved  when  the  magnitude  of  the  filter  function 
aberration  was  large  enough  spread  the  PSF  over  many  pixels.  Even  so,  it  was  readily 
apparent  that  the  spots  were  as  desired  when  viewed  in  real  time.  However,  a  more 
quantitative  result  was  desired. 

Thus,  the  approach  to  utilize  a  Wyko  400  Fizeau  PSI  interferometer  to  directly  detect  the 
phase  error  was  implemented.  In  this  approach,  the  filter  was  placed  in  the  collimated  4” 
diameter  output  of  the  interferometer,  a  longer  focal  length  lens  was  used  to  bring  the 
light  to  focus,  at  which  a  slit  was  positioned  to  pass  only  a  single  order  at  a  time.  A 
spherical  reference  mirror  was  positioned  in  the  confocal  position,  so  that  the  light  from 
the  passed  order  would  reflect  back  through  the  slit,  the  lens,  and  again  through  the  filter 
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At  this  point,  the  filter  would,  in  fact,  generate  another  whole  set  of  diffracted  orders, 
each  with  different  magnitudes  of  the  encoded  wavefront  aberration  and  each  at  a 
different  angle.  Thus,  in  order  to  directly  detect  each  diffracted  order’s  effect  required 
tilting  the  Wyko’s  reference  flat  in  order  to  ensure  that  the  wavefront  of  interest  and  the 
reference  wave  are  “parallel.” 

This  approach  allowed  us  to  take  a  single  interferogram  as  the  reference,  and  then 
subtract  this  from  the  other  orders’  interferograms  to  yield  their  phase  function. 
Subsequent  interferograms  of  the  adjacent  diffracted  orders,  containing  the  desired  Z4  or 
Z7  errors,  showed  similar  phase  structure,  but  by  allowing  the  interferometer  software  to 
reduce  this  function  minus  the  acquired  reference  into  a  Zernike  decomposition,  we  saw 
that  the  desired  filter  functions  were,  in  fact,  recorded  properly  onto  the  transparencies. 
However,  the  residual  phase  errors  in  the  substrate  were  too  large  for  the  filter  to  function 
properly.  Thus,  it  was  decided  to  remove  this  error  when  the  filters  are  implemented  in 
the  Wavefront  Sensing  System  by  immersing  the  transparencies  in  a  liquid  gate.  This 
will  eliminate  the  deformations  due  to  surface  figure  errors  in  the  printouts  which  are  the 
primary  source  of  the  distortions. 


3.3.3  Holographic  Filter  Fabrication  Lab 

Simple  printed  amplitude  filters  have  been  shown  to  be  of  very  practical  use,  but  there  are 
limitations  to  them,  primarily  efficiency  and  limited  spatial  frequency.  The  design  and 
implementation  of  many  filters  can  be  a  costly  and  slow  process  when  investigating 
numerous  filtering  approaches,  as  this  testbed  is  intended  to  be  used.  As  a  means  of 
providing  a  more  efficient  and  higher  resolution  filtering  capability,  effort  was  invested 
in  building  up  the  capabilities  of  a  hologram  fabrication  lab.  The  lab  was  assembled 
almost  exclusively  with  donated  optical  components  which  had  previously  been  used  in  a 
holography  lab.  Thus,  this  is  not  only  an  extremely  useful  addition,  but  it  is  very  cost 
effective.  The  lab  is  primarily  based  around  an  Argon  Ion  laser  operating  at  5 14nm,  and 
dichromated  gel  for  the  photosensitive  media. 

In  addition  to  being  able  to  produce  more  efficient  filters,  especially  by  using  the  selected 
orders  of  the  printed  amplitude  filters,  the  holographic  fabrication  allows  for  the 
production  of  aberration  correcting  holographic  elements.  For  example,  the  vibrating 
membrane  we  ultimately  intend  to  couple  into  the  wavefront  generator  has  some  fixed 
deformations  on  its  surface.  By  properly  writing  the  hologram,  these  fixed  errors  can  be 
removed  from  the  membrane  and  any  intervening  optical  components. 

The  ability  of  writing  holograms  has  been  demonstrated  at  514nm  and  now  at  632nm. 
Thus,  we  are  now  able  to  create  efficient  holographic  filters  at  two  wavelengths  that  can 
be  used  in  our  diffraction  limited  FTL-based  wavefront  generator  which  is  readily 
operated  at  two  wavelengths.  These  efficient  gratings  can  also  be  used  to  produce 
refractive-diffractive  hybrid  lens  systems  in  future  implementations. 
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3.4  Conclusion 

Our  goal  of  developing  a  testbed  for  advancing  this  new  Wavefront  Sensor  concept  has 
been  reached.  This  program  began  with  a  concept  and  no  additional  supporting  efforts. 
The  from-the -basics  optical  design  approach  led  to  a  loosely  toleranced  diffraction- 
limited  system  with  a  broad  wavelength  range  of  operation  and  a  massive  space 
bandwidth  product.  The  MatLAB  codes  are  in-place  to  design  and  analyze  all  new  filters 
and  system  configurations,  as  well  as  producing  functional  amplitude-only  filters.  These 
filters  can  be  utilized  to  produce  more  efficient  phase-only  filters  holographically  in  the 
hologram  fabrication  facility.  And  finally,  an  expansion  of  the  basic  Wavefront  Sensor 
form  has  been  made  in  two  directions.  First,  the  use  of  multi-order  filters  allows  for 
either  an  increase  in  the  wavefront  sensor  dynamic  range  or  its  resolution.  Second, 
applying  the  POHM  approach  allows  for  the  use  of  multiple  filters,  each  potentially 
containing  multiple  aberration  tenns,  and  each  potentially  a  controllable  SLM. 

Although  we  are  not  receiving  further  funding  through  the  Center  for  Advanced  Sensors 
for  this  research,  we  intend  to  continue  these  efforts  and  publish  further  on  our  findings. 
We  have  established  a  testbed  for  advancing  high  speed  wavefront  sensor  and  intend  to 
use  it. 
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